looker-open-source / looker_deployer

A tool to help deploy objects from one Looker instance to another

Looker Deployer

Intro

Looker Deployer (aka 'ldeploy') is a command line tool to help move Looker objects across instances. This includes Content (Looks, Dashboards, entire Spaces, etc.), Boards and Connections.

Status and Support

As of 2023, Looker Deployer is supported, but not warrantied by Google. Issues and feature requests can be reported via https://github.com/llooker/looker_deployer/issues, which will be regularly monitored and prioritized.

Requirements

In order for these commands to correctly work a few assumptions/requirements are needed for your environment:

• Python Looker Deployer requires Python 3.6-3.9
• Gazer The content deployment command makes use of gzr to automate content deployment, so you will need to have that installed and configured properly. Gazer requires an up-to-date version of ruby.

Authentication and Configuration

Looker Deployer makes use of the Looker SDK to communicate with your Looker instances. A looker.ini file is required to provide authentication. By default the tool looks for this file in your working directory but if it is named differently or in a different location you can make use of the --ini argument to specify its location. Here's an example ini file:

[dev]
base_url=https://looker-dev.company.com:19999
client_id=abc
client_secret=xyz
verify_ssl=True

[prod]
base_url=https://looker-prod.company.com:19999
client_id=abc
client_secret=xyz
verify_ssl=True

Installation

Looker Deployer is on PyPi! - You can install it with pip install looker-deployer.

Dockerfile

A looker.ini file is required for API authentication. You will have to either volume-map the ini file when you run the container, or (recommended) build an image from this one that "burns" a relevant ini file into the container.

To do this, create a directory with an looker.ini file and a Dockerfile with the following content:

FROM python:3.9-slim

RUN apt update
RUN apt -y install ruby ruby-dev
RUN gem install gazer

RUN apt -y install git 
RUN git clone https://github.com/looker-open-source/looker_deployer.git

WORKDIR /looker_deployer

COPY looker.ini . 
RUN pip install .

ENTRYPOINT ["ldeploy"]

Then simply:

docker build -t ldeploy .

You can get use: docker run ldeploy <command> <parameters> ...

Keep in mind that in order to get the output you will need to map local directories to directories in docker, so a real life usage might look like:

docker run \
  -v /local/path/to/ldeploy/settings:/ldeploy_settings \
  -v /local/path/to/ldeploy/output:/ldeploy_output \
  ldeploy content export \
    --ini /ldeploy_settins/looker.ini \
    --local-target /ldeploy_output \
    <additional parameters>

You would put your looker.ini file in /local/path/to/ldeploy/settings and the output generated by the commands would be found in /local/path/to/ldeploy/output.

Local Installation

It is recommended to set up a python virtual environment if you intend to use looker-deployer on your local machine rather than using the docker image. This will prevent other modules from creating conflicts with the looker-deployer install.

pyenv offers easy management of python versions and virtual environments.

Once pyenv is installed, install the desired version of Python and create the virtual environment named ldeploy:

pyenv install 3.8.3
pyenv virtualenv 3.8.3 ldeploy

Activate the virtual environment and install looker-deployer:

pyenv activate ldeploy
pip install looker-deployer

looker-deployer is now ready for use! Once you are finished, deactivate the virtual environment:

source deactivate

Usage

The tool is invoked with the ldeploy command, followed by the relevant sub-command. The available sub-commands are: boards, code, connections, and content.

Each sub-command can be configured with its relevant arguments, which can be reviewed with the -h or --help argument. For example:

ldeploy content -h

Content Deployment

This command makes use of gazer to either pull content from your dev Looker instance to a directory structure on your local machine or deploy content from said directory structure to a specified Looker instance. The command can work for specific sets of Looks or Dashboards or can work on entire folders - and will correctly create any folder it doesn't find in the target instance.

The content command is further divided into two subcommands: export and import

Content Export

All content deployment tasks begin by exporting a representation of your development environment's content folder tree to local disk. This is done with the export command. This directory tree is then used in subsequent import commands to import dashboards, looks, or the entire tree to another instance.

The command accepts the following arguments:

usage: ldeploy content export [-h] --env ENV [--ini INI] [--debug] --folders
                              FOLDERS [FOLDERS ...] --local-target
                              LOCAL_TARGET

optional arguments:
  -h, --help            show this help message and exit
  --env ENV             What environment to deploy from
  --ini INI             ini file to parse for credentials
  --debug               set logger to debug for more verbosity
  --folders FOLDERS [FOLDERS ...]
                        What folders to export content from
  --local-target LOCAL_TARGET
                        Local directory to store content

Content Import

Once you have exported your content from your development environment you can use the import command to bring it into your production environment.

usage: ldeploy content import [-h] --env ENV [--ini INI] [--debug]
                              [--recursive] [--target-folder TARGET_FOLDER]
                              (--folders FOLDERS [FOLDERS ...] | --dashboards DASHBOARDS [DASHBOARDS ...] | --looks LOOKS [LOOKS ...])

optional arguments:
  -h, --help            show this help message and exit
  --env ENV             What environment to deploy to
  --ini INI             ini file to parse for credentials
  --debug               set logger to debug for more verbosity
  --recursive           Should folders deploy recursively
  --target-folder TARGET_FOLDER
                        override the default target folder with a custom path
  --folders FOLDERS [FOLDERS ...]
                        Folders to fully deploy
  --dashboards DASHBOARDS [DASHBOARDS ...]
                        Dashboards to deploy
  --looks LOOKS [LOOKS ...]
                        Looks to deploy

Import base folder Support has been added for importing contenting to base folders other than Shared. There are a few important notes to go along with that:

• Users, Embed Users, and Embed Groups folders must exist before running the import, ldeploy cannot create them.
• Users, Embed Users, and Embed Groups folders must be uniquely named.
• "Shared" is the default base folder if no other is specified using "--target-folder"

Examples:

• ldeploy content export --env dev --folders 1 --local-target ./foo/bar/ <- exports the Shared folder (id 1) and all sub-folders to the directory location ./foo/bar/
• ldeploy content export --env dev --folders 5 8 --local-target ./foo/bar/ <- exports folders 5 and 8 (and all of their sub-folders) to the directory location ./foo/bar/
• ldeploy content import --env prod --folders ./foo/bar/Shared/Public <- deploys every piece of content in Shared/Public to the prod instance
• ldeploy content import --env prod --folders ./foo/bar/Shared/Public --recursive --target-folder Shared/FromDev/Public <- deploys every piece of content in Shared/Public and all sub-folders to the prod instance in the Shared/FromDev/Public folder.
• ldeploy content import --env prod --dashboards ./foo/bar/Shared/Public/Dashboard_1.json ./foo/bar/Shared/Restricted/Dashboard_2.json <- deploys Dashboard1 and Dashboard2 to their respective folders in the prod instance
• ldeploy content import --env prod --folders ./dev/Users --recursive --target-folder Users <- deploys every piece of content in dev/Users and all sub-folders to the prod instance in the Users folder
• ldeploy content import --env prod --folders "./dev/Embed Users" --recursive --target-folder "Embed Users" <- deploys every piece of content in dev/Embed Users and all sub-folders to the prod instance in the Embed Groups folder
• ldeploy content import --env prod --folders "./dev/Embed Groups" --recursive --target-folder "Embed Groups" <- deploys every piece of content in dev/Embed Groups and all sub-folders to the prod instance in the Embed Groups folder

Board Deployment

This command allows for the deployment of boards/homepages across instances. It attempts to resolve differences in dashboard/look ids across instances and confirms that the content is present before building the new board. Boards are matched by title - updating an existing board will result in the rebuilding of the relevant sections and items to prevent issues with attempting to match via title. If needing to update a board title, make use of the title-change parameter to allow the command to find the old title. The command accepts the following arguments:

usage: ldeploy boards [-h] --source SOURCE --target TARGET [TARGET ...]
                      --board BOARD [--ini INI] [--allow-partial]
                      [--title-change TITLE_CHANGE] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       which environment to source the board from
  --target TARGET [TARGET ...]
                        which target environment(s) to deploy to
  --board BOARD         which board to deploy
  --ini INI             ini file to parse for credentials
  --allow-partial       allow partial deployment of board content if not all
                        content is present on target instance?
  --title-change TITLE_CHANGE
                        if updating title, the old title to replace in target
                        environments
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy boards --source dev --target prod --board 'My Cool Board' <- deploys the board 'My Cool Board' from dev to prod
• ldeploy boards --source dev --target prod_1 prod_2 --allow-partial --board 'My Updated Title Board' --title-change 'My Cool Board' <- This deploys a board whose title has been changed from 'My Cool Board' to 'My Updated Title Board' from dev to two instances: prod_1 and prod_2. Any content not present in either prod instance will be skipped without raising any errors.

Connections Deployment

This command allows for the migration of database connections across instances. For security purposes, Looker's API does not transmit password credentials, so this command allows for the injection of these credentials from the .ini file.

The command accepts the following arguments:

usage: ldeploy connections [-h] --source SOURCE [--ini INI] --target TARGET
                           [TARGET ...] [--pattern PATTERN]
                           [--include-password] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       which environment to source the board from
  --ini INI             ini file to parse for credentials
  --target TARGET [TARGET ...]
                        which target environment(s) to deploy to
  --pattern PATTERN     regex pattern to filter which connections are deployed
  --include-password    should passwords be set from the ini file?
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy connections --source dev --target prod <- This will deploy all connections in the dev instance to the prod instance. No credentials will be included and would need to be manually added from the Looker UI.
• ldeploy connections --source dev --target prod --pattern ^bigquery --include-password <- This will deploy all connections that begin with bigquery from dev to prod and attempt to inject passwords included in the .ini file.

Code Deployment

This command will manage deployments of hub/spoke LookML code from the current Github master branch to the relevant production instance(s). This is accomplished by sending a GET request to each instances deploy endpoint and authenticating it with the webhook secret.

An overview of the relevant architecture (click to embiggen):

In order to use the code deployment tool, a code_config.yaml file is required. This file requires a list of instances - one for each client production instance. This list must include the name (used to refer to that instance in the commands), the endpoint, and the name of that instances spoke project. In addition, the config file requires an entry for the common hub project and can optionally include a list of names to instance names to exclude from hub deployments. Here's an example config:

instances:
  - name: calvin
    endpoint: https://looker-dev.company.com
    spoke_project: powered_by_spoke_ck
  - name: levis
    endpoint: https://looker-dev.company.com
    spoke_project: powered_by_spoke_lv

hub_project: powered_by_hub

The command accepts the following arguments:

usage: ldeploy code [-h] [--hub] [--spoke SPOKE [SPOKE ...]]
                    [--hub-exclude HUB_EXCLUDE [HUB_EXCLUDE ...]] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --hub                 flag to deploy hub project
  --spoke SPOKE [SPOKE ...]
                        which spoke(s) to deploy
  --hub-exclude HUB_EXCLUDE [HUB_EXCLUDE ...]
                        which projects should be ignored from hub deployment
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy code --hub <- This will deploy the hub project to all production instances
• ldeploy code --spoke foo <- This will deploy the spoke project named "foo" to the relevant instance
• ldeploy code --hub --spoke foo bar --hub-exclude bar --debug <- This will deploy the hub project to all instances except bar and then deploy the spoke projects foo and bar to their respective instances

Role Admin Settings

Due to the related nature of role settings in Looker, it is recommended to run these commands in the following order to ensure the prior information is available:

1. Model Sets
2. Permission Sets
3. Roles
4. Groups
5. Group in Group
6. Role to Group
7. User Attributes

Special Considerations:

If you have externally manged groups, you will need to ensure you update for SAML/LDAP is done prior to Group in Group. These special groups are not migrated by the code referenced above.

Model Sets Deployment

This command allows for the migration of model sets across instances.

Matching for create/update/delete will be based on the name of the setting, since IDs are auto-incremented.

The command accepts the following arguments:

usage: ldeploy model_sets [-h] --source SOURCE [--ini INI] --target TARGE [TARGET ...] [--pattern PATTERN] [--delete] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       which environment to source the model sets from
  --ini INI             ini file to parse for credentials
  --target TARGET [TARGET ...]
                        which target environment(s) to deploy to
  --pattern PATTERN     regex pattern to filter
  --delete              allows for deletion from target based on name
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy model_sets --source dev --target prod <- This will deploy all model sets in the dev instance to the prod instance.
• ldeploy model_sets --source dev --target prod --pattern ^test <- This will deploy all model sets that starts with test.

Permission Sets Deployment

This command allows for the migration of permission sets across instances.

Matching for create/update/delete will be based on the name of the setting, since IDs are auto-incremented.

The command accepts the following arguments:

usage: ldeploy permission_sets [-h] --source SOURCE [--ini INI] --target TARGE [TARGET ...] [--pattern PATTERN] [--delete] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       which environment to source the permission sets from
  --ini INI             ini file to parse for credentials
  --target TARGET [TARGET ...]
                        which target environment(s) to deploy to
  --pattern PATTERN     regex pattern to filter
  --delete              allows for deletion from target based on name
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy permission_sets --source dev --target prod <- This will deploy all permission sets in the dev instance to the prod instance.
• ldeploy permission_sets --source dev --target prod --pattern ^test <- This will deploy all permission sets that starts with test.

Roles Deployment

This command allows for the migration of roles across instances.

Matching for create/update/delete will be based on the name of the setting, since IDs are auto-incremented.

The command accepts the following arguments:

usage: ldeploy roles [-h] --source SOURCE [--ini INI] --target TARGE [TARGET ...] [--pattern PATTERN] [--delete] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       which environment to source the roles from
  --ini INI             ini file to parse for credentials
  --target TARGET [TARGET ...]
                        which target environment(s) to deploy to
  --pattern PATTERN     regex pattern to filter
  --delete              allows for deletion from target based on name
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy roles --source dev --target prod <- This will deploy all roles in the dev instance to the prod instance.
• ldeploy roles --source dev --target prod --pattern ^test <- This will deploy all roles that starts with test.

Groups Deployment

This command allows for the migration of non-externally managed groups across instances.

Matching for create/update/delete will be based on the name of the setting, since IDs are auto-incremented.

The command accepts the following arguments:

usage: ldeploy groups [-h] --source SOURCE [--ini INI] --target TARGE [TARGET ...] [--pattern PATTERN] [--delete] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       which environment to source the groups from
  --ini INI             ini file to parse for credentials
  --target TARGET [TARGET ...]
                        which target environment(s) to deploy to
  --pattern PATTERN     regex pattern to filter
  --delete              allows for deletion from target based on name
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy groups --source dev --target prod <- This will deploy all groups in the dev instance to the prod instance.
• ldeploy groups --source dev --target prod --pattern ^test <- This will deploy all groups that starts with test.

Group in Group (i.e. Groups Hierarchy) Deployment

This command allows for the migration of groups belonging to other groups across instances.

Matching for create or delete will be based on the name of the setting, since IDs are auto-incremented.

The command accepts the following arguments:

usage: ldeploy group_in_group [-h] --source SOURCE [--ini INI] --target TARGE [TARGET ...] [--pattern PATTERN] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       which environment to source the group hierarchy from
  --ini INI             ini file to parse for credentials
  --target TARGET [TARGET ...]
                        which target environment(s) to deploy to
  --pattern PATTERN     regex pattern to filter
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy group_in_group --source dev --target prod <- This will deploy all group in groups in the dev instance to the prod instance.
• ldeploy group_in_group --source dev --target prod --pattern ^test <- This will deploy all group in groups that starts with test.

Role to Group Deployment

This command allows for the migration of roles associated to groups across instances.

Matching for create or update will be based on the name of the setting, since IDs are auto-incremented.

The command accepts the following arguments:

usage: ldeploy role_to_group [-h] --source SOURCE [--ini INI] --target TARGE [TARGET ...] [--pattern PATTERN] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       which environment to source the role to group from
  --ini INI             ini file to parse for credentials
  --target TARGET [TARGET ...]
                        which target environment(s) to deploy to
  --pattern PATTERN     regex pattern to filter
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy role_to_group --source dev --target prod <- This will deploy all roles to group in the dev instance to the prod instance.
• ldeploy role_to_group --source dev --target prod --pattern ^test <- This will deploy all roles to groups that starts with test.

User Attributes Deployment

This command allows for the migration of non-system managed user attributes across instances (i.e. id, first_name, last_name, email).

Matching for create/update/delete will be based on the name of the setting, since IDs are auto-incremented. This includes assignments made for user attributes made by group names.

The command accepts the following arguments:

usage: ldeploy user_attributes [-h] --source SOURCE [--ini INI] --target TARGE [TARGET ...] [--pattern PATTERN] [--delete] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       which environment to source the user attributes from
  --ini INI             ini file to parse for credentials
  --target TARGET [TARGET ...]
                        which target environment(s) to deploy to
  --pattern PATTERN     regex pattern to filter
  --delete              allows for deletion from target based on name
  --debug               set logger to debug for more verbosity

Examples:

• ldeploy user_attributes --source dev --target prod <- This will deploy all user attributes in the dev instance to the prod instance.
• ldeploy user_attributes --source dev --target prod --pattern ^test <- This will deploy all user attributes that starts with test.

Development

This project makes use of pipenv to manage dependencies. Follow the installation instructions. It is recommended to use pyenv to manage installing python versions.

Once pipenv has been installed, clone this repo, cd into it, and invoke pipenv install --ignore-pipfile

looker_deployer's Issues

Bump python version requirements

Currently the project is being locked at using python version 3.7 - see here.

In addition in setup.py you have a range - see here

Would be nice to

a) keep both in sync , including docs - see here
b) allow/ use a newer version of python - v3.11

Model configuration

Hello Team

I would like to configure model automatically through the pipeline once it is deployed on PROD from DEV. Is this something looker_deployer is designed to support in some form ?

Problem with the import

Hello everyone,

After running ldeploy content import --env dev --recursive --folders ./dashboards/Shared/

Looker deployer successfully imports dashboards that are in the shared folder, but for dashboards that are in other folders, I get an error :

{"levelname": "INFO", "module": "deploy_content", "funcName": "deploy_space", "message": "Attemting Recursion of children folders", "children_folders": ["./dashboards/Shared/Folder_1\\", "./dashboards/Shared/Folder_2\\", "./dashboards/Shared/Folder_3\\"], "timestamp": "2023-10-19T14:29:02.450314Z"} {"levelname": "WARNING", "module": "deploy_content", "funcName": "create_or_return_space", "message": "No folders found. Creating folder now", "timestamp": "2023-10-19T14:29:03.519255Z"} Traceback (most recent call last): etc...

Lookerdeployer don't want to create folders apparently. What can I do ?

ERROR: create_query(...) returned {:message=>"The resource you're looking for could not be found"}

In the process of migrating Looker instances, I used ldeploy content export to pull down all the Users' folders, and then used ldeploy content import to copy the dashboards and looks into the User's corresponding folders in the new instance. ~90% of the time it worked fine, but for a few of the Looks, I got the above issue. The closest I could find online was this post from six months ago, implying it has to do with the api change from 3.1 to 4.0 https://community.looker.com/open-source-projects-78/n0t-able-to-use-gazer-import-commands-after-looker-version-migrate-to-22-4-29753.

The full command i'm running is ldeploy content import --env prod --ini looker.ini --looks "./users_folder_export/Users/<Firstname Lastname>/Look_80_<Look Name>.json" --target-folder "Users/<User Name>, though removing the --target folder gives the same error

Difficulty setting up looker_deployer in pyenv, virtualenv

New Looker customer - working through setting up looker_deployer on Ubuntu 18.04

Is there anyone successfully deploying looker_deployer under pyenv, virtualenv who is willing to review/zoom call to assist?

Having difficulty getting the pyenv, virtual env setup correct to enable looker_deployer to run ---- getting the following errors:

when within virtualenv (python 3.8.3) --- gives back an ImportError: cannot import name 'get_args''

ldeploy -h

Traceback (most recent call last):
File "/usr/local/bin/ldeploy", line 7, in
from looker_deployer.cli import main
File "/home/ubuntu/.local/lib/python3.6/site-packages/looker_deployer/cli.py", line 2, in
from looker_deployer.commands import deploy_boards, deploy_code, deploy_connections
File "/home/ubuntu/.local/lib/python3.6/site-packages/looker_deployer/commands/deploy_boards.py", line 2, in
from looker_sdk import models
File "/home/ubuntu/.local/lib/python3.6/site-packages/looker_sdk/init.py", line 27, in
from looker_sdk.rtl import serialize
File "/home/ubuntu/.local/lib/python3.6/site-packages/looker_sdk/rtl/serialize.py", line 45, in
import cattr
File "/home/ubuntu/.local/lib/python3.6/site-packages/cattr/init.py", line 1, in
from .converters import Converter, GenConverter, UnstructureStrategy
File "/home/ubuntu/.local/lib/python3.6/site-packages/cattr/converters.py", line 18, in
from ._compat import (
File "/home/ubuntu/.local/lib/python3.6/site-packages/cattr/_compat.py", line 31, in
from typing import get_args, get_origin # NOQA
ImportError: cannot import name 'get_args'

FR - Support for promoting user attributes between instances

It sometimes occurs that a bit of LookML and/or a dashboard setting contains a reference to a user attribute. In those cases, the attribute must exist in all instances hosting that code and/or content. With the Looker Deplopyer, it is possible to promote content, but the user attribute which powers that content will need to be made manually. It would be useful to be able to "copy" a user attribute between instances.

I would think a first iteration, and perhaps all that would really be necessary in the kinds of deployments using LDeploy, would be to move the "first page" of settings to the new instance. Group assignments of values or individual assignments would still need to be done manually. A default value, if it exists, could be promoted to the new instance.

Changed Slugs when Syncing Dashboards

I wrote a bash script that runs in Github Actions and uses Looker Deployer to sync dashboards and looks from our dev Looker instance to our prod Looker instance. We are not syncing the entire /Shared folder every time, we are syncing subfolders.

Internally, Looker Deployer is running a series of Gazer commands to accomplish that syncing. I am using the "deploy content" command in Looker Deployer. You can see the gazer commands that it runs here:
https://github.com/llooker/looker_deployer/blob/master/looker_deployer/commands/deploy_content.py#L76
As you can see, it runs gazer with --force.

When we move an existing dashboard to a new subfolder, when we are syncing using --force, then **it seems like ** Gazer deletes the existing dashboard and sync a new copy. That delete moved the existing dashboard to the Trash folder. When this happens there is nothing listed in the "deleted by" column of some of the deleted dashboards in our Trash folder on our prod Looker instance, but I can't always duplicate this behavior. Sometimes Gazer just creates a new dashboard in the new folder without deleting the old dashboard, even though we are only moving the dashboards between different child folders of the parent folder we are syncing.

Gazer syncs dashboards **using a new slug ** because it detects that there is a dashboard with a duplicate slug in the Trash/in the other folder on the production Looker instance. You can see the code that does that here:
https://github.com/pixability/gazer/blob/master/lib/gzr/commands/dashboard/import.rb#L141

As a result, the dashboard slugs are out of sync sometimes and that breaks links in our dashboards on the prod instance.

Another note here: Since the team that runs the sync jobs doesn't have admin access to our prod Looker instance, we can't ask them to make sure that dashboards are hard deleted. We need to have someone from the dev team manually intervene.

If you seen this before, and am I accurately understanding what is happening? Can you tell me what the expected behavior is, and whether there are any workarounds for this?

Getting an undefined: method 'space error

Has anything changed for looker_deployer for api 4.0? I am getting this error when I try to export my dashboards from the dev environment:
options: {"debug"=>true, "host"=>"revegydev.looker.com", "port"=>"19999", "ssl"=>true, "verify_ssl"=>false, "timeout"=>60, Removed Client ID and Secret
"dir"=>"content41122/Shared"}
API 3.1 available? true
connecting to ["api_endpoint=>https://revegydev.looker.com:19999/api/4.0", "connection_options=>{:ssl=>{:verify=>false}, :request=>{:timeout=>60}}", "user_agent=>Gazer 0.2.43", "client_id=>Remove", "client_secret=>*********"]
check for connectivity: true
verify authentication: true
logout
Traceback (most recent call last):
18: from /usr/local/bin/gzr:23:in <main>' 17: from /usr/local/bin/gzr:23:in load'
16: from /var/lib/gems/2.5.0/gems/gazer-0.2.43/exe/gzr:36:in <top (required)>' 15: from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in start'
14: from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch' 13: from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in invoke_command'
12: from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run' 11: from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:238:in block in subcommand'
10: from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:115:in invoke' 9: from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch'
8: from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in invoke_command' 7: from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run'
6: from /var/lib/gems/2.5.0/gems/gazer-0.2.43/lib/gzr/commands/space.rb:80:in export' 5: from /var/lib/gems/2.5.0/gems/gazer-0.2.43/lib/gzr/commands/space/export.rb:49:in execute'
4: from /var/lib/gems/2.5.0/gems/gazer-0.2.43/lib/gzr/modules/session.rb:207:in with_session' 3: from /var/lib/gems/2.5.0/gems/gazer-0.2.43/lib/gzr/commands/space/export.rb:81:in block in execute'
2: from /var/lib/gems/2.5.0/gems/gazer-0.2.43/lib/gzr/commands/space/export.rb:87:in process_space' 1: from /var/lib/gems/2.5.0/gems/gazer-0.2.43/lib/gzr/modules/space.rb:65:in query_space'
/var/lib/gems/2.5.0/gems/looker-sdk-0.0.7/lib/looker-sdk/client/dynamic.rb:105:in method_missing': undefined method space' for #LookerSDK::Client:0x000055fccc55f338 (NoMethodError)

Here is the line that I am running: ldeploy content export --env dev --folders 212 --local-target ./content41122/ --debug

Error: Getting "Requires authentication" error while exporting content.

Hello Team,
I am getting “Requires authentication” error while exporting content. I have checked looker.ini file in that all credentials and base_url’s are correct and working. Earlier it was working fine. I am getting this error since last week. Are there any changes/modifications done in the code or do I need to add anything in my export command ? Could you please help me with this it is a bit urgent. Also I have attached logs for your reference:

File "/usr/local/lib/python3.7/site-packages/looker_sdk/rtl/auth_session.py",line87,in_get_token self._login(transport_options)
File "/usr/local/lib/python3.7/site-packages/looker_sdk/rtl/auth_session.py", line 153, in _login
transport_options=transport_options,
File "/us/local/lib/python3.7/site-packages/looker_sdk/rtl/auth_session.py",line248,in _ok raise error.SDKError(response.value)
looker_sdk.error.SDKError: b'{"message": "Requires
authentication.", "documentation_url":"https://cloud.google.com/looker/docs/"}'

Connexion deployment is not functionning.

Hello everyone,

I have tried these 2 commands :

ldeploy connections --source dev --target prod
ldeploy connections --source dev --target prod --include-password

In the first case I have this issue : SDK error.
In the second case I have this issue : KeyError: 'Databases' (db_config = parse_ini.read_ini(args.ini)["Databases"])

Do you know what can I do ?

Dockerfile results in `method_missing` error

I am currently deploying looker_deployer at a client and am following the README to create the Docker image where commands can be run. Unfortunately I keep getting the same error. I see issue #79 where a similar error message was reported, the solution in this case was to update the gazer version (this version is now outdated so this approach is not possible). Any ideas why I am getting this "undefined method `space'" error or steps I can take to debug?

Steps to repeat:

1. Initialise looker.ini with a "dev" target.
2. Run:

docker build -t ldeploy .
docker run ldeploy content export --env dev --ini looker.ini --debug --folders 12 --local-target ./export

Resulting logs:

{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "main", "message": "ini file", "ini": "looker.ini", "timestamp": "2023-04-03T13:47:19.176112Z"}
{"levelname": "INFO", "module": "deploy_content_export", "funcName": "main", "message": "Exporting content", "env": "dev", "folders": ["12"], "dashboards": null, "looks": null, "local_target": "./export", "timestamp": "2023-04-03T13:47:19.176343Z"}
{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "recurse_folders", "message": "recursive folder crawl status", "current_id": "12", "folder_name": "Marketing", "current_list": ["Marketing"], "timestamp": "2023-04-03T13:47:20.822746Z"}
{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "recurse_folders", "message": "going for recursion", "parent_id": "1", "timestamp": "2023-04-03T13:47:20.822962Z"}
{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "recurse_folders", "message": "recursive folder crawl status", "current_id": "1", "folder_name": "Shared", "current_list": ["Marketing", "Shared"], "timestamp": "2023-04-03T13:47:21.407126Z"}
{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "send_export", "message": "folder_list", "folder_id": "12", "list": ["Shared", "Marketing"], "timestamp": "2023-04-03T13:47:21.407325Z"}
/var/lib/gems/2.7.0/gems/looker-sdk-0.1.2/lib/looker-sdk/client/dynamic.rb:106:in `method_missing': undefined method `space' for #<LookerSDK::Client:0x000055e2e8ce4ed0> (NoMethodError)
        from /var/lib/gems/2.7.0/gems/gazer-0.2.59/lib/gzr/modules/space.rb:65:in `query_space'
        from /var/lib/gems/2.7.0/gems/gazer-0.2.59/lib/gzr/commands/space/export.rb:89:in `process_space'
        from /var/lib/gems/2.7.0/gems/gazer-0.2.59/lib/gzr/commands/space/export.rb:83:in `block in execute'
        from /var/lib/gems/2.7.0/gems/gazer-0.2.59/lib/gzr/modules/session.rb:249:in `with_session'
        from /var/lib/gems/2.7.0/gems/gazer-0.2.59/lib/gzr/commands/space/export.rb:51:in `execute'
        from /var/lib/gems/2.7.0/gems/gazer-0.2.59/lib/gzr/commands/space.rb:82:in `export'
        from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
        from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
        from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:116:in `invoke'
        from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:243:in `block in subcommand'
        from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
        from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
        from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
        from /var/lib/gems/2.7.0/gems/gazer-0.2.59/exe/gzr:36:in `<top (required)>'
        from /usr/local/bin/gzr:23:in `load'
        from /usr/local/bin/gzr:23:in `<main>'
options: {"debug"=>true, "host"=>"<REDACTED>", "port"=>"19999", "ssl"=>true, "verify_ssl"=>true, "timeout"=>60, "force"=>false, "persistent"=>false, "client_id"=>"<REDACTED>", "client_secret"=>"<REDACTED>", "dir"=>"export/Shared"}
using options ["debug=>true", "host=><REDACTED>", "port=>19999", "ssl=>true", "verify_ssl=>true", "timeout=>60", "force=>false", "persistent=>false", "client_id=><REDACTED>", "dir=>export/Shared"]
API current_version 4.0
API versions ["3.0", "3.1", "4.0"]
connecting to ["api_endpoint=>https://<REDACTED>:19999/api/4.0", "connection_options=>{:ssl=>{:verify=>true, :verify_mode=>3}, :request=>{:timeout=>60}}", "user_agent=>Gazer 0.2.59", "client_id=><REDACTED>", "client_secret=><REDACTED>"]
check for connectivity: true
verify authentication: true
logout

Versions:

root@1b1a4454b1f5:/looker_deployer# ruby -v
ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux-gnu]
root@1b1a4454b1f5:/looker_deployer# gem -v
3.2.5
root@1b1a4454b1f5:/looker_deployer# gem list | grep "gazer"
gazer (0.2.59)
root@1b1a4454b1f5:/looker_deployer# gem list | grep "looker"
looker-sdk (0.1.2)

Dockerfile:

FROM python:3.9-slim

RUN apt update
RUN apt -y install ruby ruby-dev
RUN gem install gazer

RUN apt -y install git 
RUN git clone https://github.com/looker-open-source/looker_deployer.git --branch looker-deployer-v0.3.9

WORKDIR /looker_deployer

COPY looker.ini . 
RUN pip install -e .

ENTRYPOINT ["ldeploy"]

testing

testing123

Unable to Import Content

I am trying to migrate a dashboard from one instance to another. I was able to export the Shared folder, but when I try to deploy the content to another instance I get the following error:

logout
ERROR: Connection Failed.
Did you specify the --no-ssl option for an ssl secured server?
You may need to use --port=443 in some cases as well.

I double-checked the client ID/secret as well as the port. But I am still having issues.

Thanks,
Drew

ValueError: Unsupported type: _ForwardRef('DashboardBase'). Register a structure hook for it.

I'm trying to export from my prod environment as follows:

ldeploy content export --env prod --folders 1 --local-target .

And I'm getting the following error:

ValueError: Unsupported type: _ForwardRef('DashboardBase'). Register a structure hook for it.

Call Stack:

{"levelname": "INFO", "module": "deploy_content_export", "funcName": "main", "message": "Exporting content", "env": "prod", "folders": ["1"], "local_target": ".", "timestamp": "2020-11-20T04:26:25.896895Z"}
Traceback (most recent call last):
  File "/Users/umair/.pyenv/versions/3.6.0/bin/ldeploy", line 8, in <module>
    sys.exit(main())
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/looker_deployer/cli.py", line 25, in main
    args.func(args)
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/looker_deployer/commands/deploy_content_export.py", line 92, in main
    send_export(args.folders, args.local_target, args.env, args.ini, sdk, args.debug)
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/looker_deployer/commands/deploy_content_export.py", line 66, in send_export
    folder_list = recurse_folders(fid, folder_list, sdk, debug)
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/looker_deployer/commands/deploy_content_export.py", line 48, in recurse_folders
    space = sdk.space(str(folder_id))
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/looker_sdk/sdk/api31/methods.py", line 7076, in space
    transport_options=transport_options,
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/looker_sdk/rtl/api_methods.py", line 152, in get
    return self._return(response, structure)
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/looker_sdk/rtl/api_methods.py", line 101, in _return
    ret = self.deserialize(data=value, structure=structure)  # type: ignore
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/looker_sdk/rtl/serialize.py", line 79, in deserialize
    data, structure
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/cattr/converters.py", line 192, in structure
    return self._structure_func.dispatch(cl)(obj, cl)
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/cattr/converters.py", line 314, in structure_attrs_fromdict
    dispatch(type_)(val, type_) if type_ is not None else val
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/cattr/converters.py", line 387, in _structure_union
    return self._structure_func.dispatch(other)(obj, other)
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/cattr/converters.py", line 327, in _structure_list
    for e in obj
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/cattr/converters.py", line 327, in <listcomp>
    for e in obj
  File "/Users/umair/.pyenv/versions/3.6.0/lib/python3.6/site-packages/cattr/converters.py", line 255, in _structure_default
    raise ValueError(msg)

Send a descriptive user agent string for Looker API calls

For Looker API calls, set a custom user agent string to something like looker_deployer-<version>

SSL Certificate could not be verified

hello everyone, I'm trying to connect to my gcp instance in looker deployer but it doesn't work.

After putting in my looker.ini
[prod] base_url=https://xxx.cloud.looker.com:443 client_id=yyy client_secret=zzz verify_ssl=True

and executing the following code :

ldeploy content export --env prod --folders 1 --local-target ./dashboards

I obtain the error :

{"levelname": "INFO", "module": "deploy_content_export", "funcName": "main", "message": "Exporting content", "env": "prod", "folders": ["1"], "dashboards": null, "looks": null, "local_target": "./dashboards", "timestamp": "2023-09-27T09:49:37.908669Z"} ERROR: SSL Certificate could not be verified Do you need the --no-verify-ssl option or the --no-ssl option?

What can I do ?

I just find the solution by changing the ini file;

Looker Import Target Folder

Hi,

Is it possible to implement a feature that allows for importing to a non-Shared folder? Currently, the target folder must start with "Shared," but I would like to import for example user 14's space in staging to user 14's space in prod. Thanks!

looker_deployer is not working

I checked with both the methods
1 creating dockerfile and run it and export content
2) setting up the local environment and then try to export content from dev env

But I am failing during exporting the content from dev looker instance with below error.

Kindly suggest why its failing as I have setup locally as per the Readme file.

1. ubuntu@ip-172-31-6-67:~/Looker$ sudo docker run ldeploy -v /local/path/to/ldeploy/settings:/ldeploy_settings -v /local/path/to/ldeploy/output:/ldeploy_output ldeploy content export --ini /ldeploy_settings/looker.ini --local-target /ldeploy_output -v --boards Boards
   usage: ldeploy [-h] [-v]
   {boards,code,connections,content,permission_sets,model_sets,roles,groups,group_in_group,role_to_group,user_attributes}
   ...
   ldeploy: error: argument {boards,code,connections,content,permission_sets,model_sets,roles,groups,group_in_group,role_to_group,user_attributes}: invalid choice: '/local/path/to/ldeploy/settings:/ldeploy_settings' (choose from 'boards', 'code', 'connections', 'content', 'permission_sets', 'model_sets', 'roles', 'groups', 'group_in_group', 'role_to_group', 'user_attributes')

2. ldeploy content export --env dev --ini looker.ini --folders Folders --local-target ./Looker
   {"levelname": "INFO", "module": "deploy_content_export", "funcName": "main", "message": "Exporting content", "env": "dev", "folders": ["Folders/shared folders/Test123"], "dashboards": null, "looks": null, "local_target": "./Looker", "timestamp": "2023-11-29T06:52:44.762856Z"}
   Traceback (most recent call last):
   File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_sdk/rtl/serialize.py", line 66, in deserialize
   data = json.loads(data)
   File "/usr/lib/python3.10/json/init.py", line 346, in loads
   return _default_decoder.decode(s)
   File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
   obj, end = self.raw_decode(s, idx=_w(s, 0).end())
   File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
   raise JSONDecodeError("Expecting value", s, err.value) from None
   json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_sdk/rtl/api_methods.py", line 90, in _return
sdk_error = self.deserialize(data=value, structure=error.SDKError) # type: ignore
File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_sdk/rtl/serialize.py", line 68, in deserialize
raise DeserializeError(f"Bad json {ex}")
looker_sdk.rtl.serialize.DeserializeError: Bad json Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/.local/bin/ldeploy", line 8, in
sys.exit(main())
File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_deployer/cli.py", line 46, in main
args.func(args)
File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_deployer/commands/deploy_content_export.py", line 164, in main
send_export(
File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_deployer/commands/deploy_content_export.py", line 116, in send_export
folder_list = recurse_folders(fid, folder_list, sdk, debug)
File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_deployer/commands/deploy_content_export.py", line 96, in recurse_folders
space = sdk.folder(str(folder_id))
File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_sdk/sdk/api40/methods.py", line 5888, in folder
self.get(
File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_sdk/rtl/api_methods.py", line 149, in get
return self._return(response, structure)
File "/home/ubuntu/.local/lib/python3.10/site-packages/looker_sdk/rtl/api_methods.py", line 92, in _return
raise error.SDKError(value)
looker_sdk.error.SDKError:

<title>Looker Not Found (404)</title>

<!-- @@@@@@@@@@@@@ FAVICONS @@@@@@@@@@@@@ -->

<link rel="apple-touch-icon-precomposed" sizes="57x57" href=https://wwwstatic-b.lookercdn.com/favicon/apple-touch-icon-57x57.png />
<link rel="apple-touch-icon-precomposed" sizes="114x114" href=https://wwwstatic-c.lookercdn.com/favicon/apple-touch-icon-114x114.png />
<link rel="apple-touch-icon-precomposed" sizes="72x72" href=https://wwwstatic-d.lookercdn.com/favicon/apple-touch-icon-72x72.png />
<link rel="apple-touch-icon-precomposed" sizes="144x144" href=https://wwwstatic-a.lookercdn.com/favicon/apple-touch-icon-144x144.png />
<link rel="apple-touch-icon-precomposed" sizes="60x60" href=https://wwwstatic-b.lookercdn.com/favicon/apple-touch-icon-60x60.png />
<link rel="apple-touch-icon-precomposed" sizes="120x120" href=https://wwwstatic-c.lookercdn.com/favicon/apple-touch-icon-120x120.png />
<link rel="apple-touch-icon-precomposed" sizes="76x76" href=https://wwwstatic-d.lookercdn.com/favicon/apple-touch-icon-76x76.png />
<link rel="apple-touch-icon-precomposed" sizes="152x152" href=https://wwwstatic-a.lookercdn.com/favicon/apple-touch-icon-152x152.png />
<link rel="icon" type="image/png" href=https://wwwstatic-b.lookercdn.com/favicon/favicon-196x196.png sizes="196x196" />
<link rel="icon" type="image/png" href=https://wwwstatic-c.lookercdn.com/favicon/favicon-96x96.png sizes="96x96" />
<link rel="icon" type="image/png" href=https://wwwstatic-d.lookercdn.com/favicon/favicon-32x32.png sizes="32x32" />
<link rel="icon" type="image/png" href=https://wwwstatic-a.lookercdn.com/favicon/favicon-16x16.png sizes="16x16" />
<link rel="icon" type="image/png" href=https://wwwstatic-b.lookercdn.com/favicon/favicon-128.png sizes="128x128" />
<meta name="application-name" content="Looker"/>
<meta name="msapplication-TileColor" content="#FFFFFF" />
<meta name="msapplication-TileImage" content=https://wwwstatic-c.lookercdn.com/favicon/mstile-144x144.png />
<meta name="msapplication-square70x70logo" content=https://wwwstatic-d.lookercdn.com/favicon/mstile-70x70.png />
<meta name="msapplication-square150x150logo" content=https://wwwstatic-a.lookercdn.com/favicon/mstile-150x150.png />
<meta name="msapplication-wide310x150logo" content=https://wwwstatic-b.lookercdn.com/favicon/mstile-310x150.png />
<meta name="msapplication-square310x310logo" content=https://wwwstatic-c.lookercdn.com/favicon/mstile-310x310.png />

<style type="text/css">
    body {
        background-color: #2e343f;
        color: white;
        height: auto;
        font-family: Open Sans, Helvetica, Arial, sans-serif;
    }
    .message {
        width: 100%;
        max-width: 760px;
        margin: 0 auto;
        margin-top: 135px;
        text-align: center;
    }
    h2, h3 {
        font-weight: normal;
    }
    a {
        color: white;
    }
</style>

<div class="message">

    <img width="210" height="84" src=https://wwwstatic-a.lookercdn.com/logos/looker_all_white.svg alt="Looker">

    <h1>Looker is unavailable.</h1>

    <h2>If you typed in a URL, double-check the spelling.</h2>
    <h2>This may also be due to a temporary condition such as an outage, <a href=https://docs.looker.com/relnotes/hosted-maintenance-hours>scheduled maintenance</a> or upgrade.</h2>
    <br>
    <h3>
      If this message persists or you have any concerns, <br> contact us from
      <a href=https://help.looker.com/>help.looker.com</a> and we'll respond promptly.
    </h3>

</div>

Exporting specific content (not entire folders)

Description

I would like to export a specific dashboard to my local directory.

In the README.md, it states

This command makes use of gazer to either pull content from your dev Looker instance to a directory structure on your local machine or deploy content from said directory structure to a specified Looker instance. The command can work for specific sets of Looks or Dashboards or can work on entire folders - and will correctly create any folder it doesn't find in the target instance.

However, within the export command, I can only see the --folders argument. I would like to export specific content i.e. an individual dashboard. Given the import dashboard command is possible, I would have assumed that the opposite capability is possible i.e. export dashboard.

ldeploy import issue

Hi Team, I am working on ldeploy POC for my organization and I am pretty excited about this tool. When this tool is productionized, it will definitely add lot of values and benefits for our team.

I am able to successfully export the contents from one instance but when I try to import this to other instance, I keep getting the following error and the dashboard is being created in the new instance with only filters but I don't see any Tiles. Am I missing something here?

ERROR: The resource you're looking for could not be found
Error creating query({
"view": "customer_demo",
"fields": [

ERROR: Operation requires API v3.1, which is not available from this host

I'm trying to download content from our instance, and initially, I was getting authentication errors. I uninstalled the looker-deployer package and reinstalled it, and now I get this error:

ERROR: Operation requires API v3.1, which is not available from this host

Is there an API 4.0 version available? I tried to search the code for the api reference, but I couldn't find anything, maybe it's in the gazer project? I didn't manually install gazer, is there a way to update that, if the API is updated there?

What's the best way to proceed on API 4.0 so I can download content from our Looker instance? Thanks!

Content import for dashboards and looks broken

It looks like this PR: https://github.com/looker-open-source/looker_deployer/pull/91/files added the ability to add a target folder where it is going to import the content to. This then causes on

looker_deployer/looker_deployer/commands/deploy_content.py

Line 215 in 3065a4f

a, b, c = dirs.partition(target_base)

to error with: TypeError: must be str, not NoneType when doing dashboard or look imports. This is due to those deploy_content calls not being updated to include the target_base parameter (ie:

looker_deployer/looker_deployer/commands/deploy_content.py

Lines 266 to 268 in 3065a4f

	deploy_content("dashboard", new_dash_path, sdk, env, ini, debug)
	else:
	deploy_content("dashboard", dash, sdk, env, ini, debug)

).

Will submit PR shortly with fix.

Feature Request: Copy Folder Permissions when generating a new folder

Or ... am I using this wrong?

I noticed that when I moved replicated a bunch of content to a new instance and the folders had to be created, none of the groups (which I had transferred) had access to the folders. It seems like they just inherited the permissions from the shared (parent) folder.

Getting error while using Looker Deployer 0.3.9

Using Gazer V 0.2.54 (also tried with V0.2.59 & 0.2.58) and Looker_Deployer V0.3.9. Getting the following error started this week only:

/var/lib/gems/2.7.0/gems/looker-sdk-0.1.4/lib/looker-sdk/client.rb:475:in merge_content_type_if_body': uninitialized constant Faraday::UploadIO (NameError) from /var/lib/gems/2.7.0/gems/looker-sdk-0.1.4/lib/looker-sdk/client/dynamic.rb:149:in invoke_remote'
from /var/lib/gems/2.7.0/gems/looker-sdk-0.1.4/lib/looker-sdk/client/dynamic.rb:107:in method_missing' from /var/lib/gems/2.7.0/gems/gazer-0.2.54/lib/gzr/modules/dashboard.rb:99:in update_dashboard'
from /var/lib/gems/2.7.0/gems/gazer-0.2.54/lib/gzr/commands/dashboard/import.rb:168:in sync_dashboard' from /var/lib/gems/2.7.0/gems/gazer-0.2.54/lib/gzr/commands/dashboard/import.rb:66:in block (2 levels) in execute'
from /var/lib/gems/2.7.0/gems/gazer-0.2.54/lib/gzr/modules/filehelper.rb:88:in read_file' from /var/lib/gems/2.7.0/gems/gazer-0.2.54/lib/gzr/commands/dashboard/import.rb:54:in block in execute'
from /var/lib/gems/2.7.0/gems/gazer-0.2.54/lib/gzr/modules/session.rb:249:in with_session' from /var/lib/gems/2.7.0/gems/gazer-0.2.54/lib/gzr/commands/dashboard/import.rb:50:in execute'
from /var/lib/gems/2.7.0/gems/gazer-0.2.54/lib/gzr/commands/dashboard.rb:78:in import' from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in run'
from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in invoke_command' from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in dispatch'
from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:116:in invoke' from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:243:in block in subcommand'
from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in run' from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in invoke_command'
from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in dispatch' from /var/lib/gems/2.7.0/gems/thor-1.2.1/lib/thor/base.rb:485:in start'
from /var/lib/gems/2.7.0/gems/gazer-0.2.54/exe/gzr:36:in <top (required)>' from /usr/local/bin/gzr:[23](https://github.com/sharphealthcare/looker-clinical/actions/runs/4557253637/jobs/8083675952#step:4:24):in load'
from /usr/local/bin/gzr:23:in `

'
Modifying existing dashboard 35 Emergency Department NDNQI Statistics in space 10

Looker not recognize the credentials.

Using ldeploy (looker deployer) migrated couple dashboards, then connected git repos for LookML dashboards. After this connection, ldeploy fails to connect to looker and throws "<title>Looker Not Found (404)</title>" error. Tried with different credentials and validated that base_url is correct.

Any suggestions, on troubleshooting this issue.

Thanks!

test issue

testing

Update API version to 4.0

Currently, deployer is using api 3.1 which is going to be deprecated on June 20. Please update it to use version 4

execution expired (Faraday::ConnectionFailed)

Hello, I have a strange situation in the Google Cloud running ldeploy.
First, my build:

Starting with Ubuntu 20.04
apt install -y python3-dev
apt install -y python3-pip
apt install -y ruby
gem install gazer
pip3 install looker-sdk
pip3 install looker-deployer

When I do this build in my own personal CGP account "ldeploy content export folder" works perfectly fine.

ldeploy content export --ini /u01/admin/conf/looker_environments.ini --env dev --folders 1 --local-target /u01/admin/data

When I do this build in my company's GCP account, the same command ends in a timeout:
ldeploy-export-error.txt

So I am suspecting some kind of firewall issue. Egress is wide open.
Do I need to open specific ports for ingress for ldeploy to work from GCP? I don't see anything in the doc with respect to this.

ldeployer content export import does not migrate existing schedule information

ldeployer content export import does migrate existing schedule information

Spelling error in README

ldeploy_settins => ldeploy_settings

KeyError:‘results’ in deploy_code file stopping deployment -> Fix to try-except block?

@drstrangelooker I am getting a KeyError:‘results’ when trying to deploy hub code for just our QA stack.I’ve narrowed down the line of code to come from the deploy_code py file here (see pictured)

Can we submit a hotfix to update this line of code to skip this for QA via a try-except block instead?

Also, can you tell me what “results” is supposed to be looking for? I think this will always error for QA but we don’t want it to stop our hub code deployment.

After export, import doesn't seem to recognize the location of the data stored previously.

Hi everyone, new to docker and looker deployer, I tried running this command below using docker, I have the looker.ini file and Dockerfile defined.

docker run -it ldeploy content export --env dev --ini ./looker.ini --debug --folders 24 --local-target ./clement/test/

But when i try to import, i encounter an error saying./test/ file not found exception

Currently running on
python v3.7.3
looker-deployer v0.3.5
gzr v0.2.54

ldeploy on RedHat 8 - tries to validate SSL even though set to False in INI file

Hi. I am working with looker-deployer on Red Hat 8.

It is failing on the following error

 1: from /usr/local/rvm/rubies/ruby-2.7.2/lib/ruby/2.7.0/net/protocol.rb:44:in `ssl_socket_connect'

/usr/local/rvm/rubies/ruby-2.7.2/lib/ruby/2.7.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (EE certificate key too weak) (OpenSSL::SSL::SSLError)

Appears to validate the SSL certificate in spite of looker.ini settings to False

Any help or insights would be appreciated

Dockerfile exits right after load

Running into issue where docker file exits immediatley after loading. Both the Dockerfile and the pre-built image on dockerhub

.

Getting error while trying to deploy dashboard

Getting following error while trying to deploy dashboard through github actions. We have self hosted github runners based on python:3.8-slim build with gazer and looker-deployer installed. We had no problems in exporting the data. If we run the same process locally, with everything being same, we are not getting any errors.

ldeploy content import --env dev --ini /app/looker.ini --debug --dashboards "deployfolder/Shared/Clinical/Clinical Effectiveness/Dashboard_9_CMS OP-22.json"
{"levelname": "DEBUG", "module": "deploy_content", "funcName": "main", "message": "ini file", "ini": "/app/looker.ini", "timestamp": "2022-09-22T17:48:51.109751Z"}
{"levelname": "DEBUG", "module": "deploy_content", "funcName": "send_content", "message": "Deploying dashboards", "dashboards":
["deployfolder/Shared/Clinical/Clinical Effectiveness/Dashboard_9_CMS OP-22.json"], "timestamp": "2022-09-22T17:48:51.110706Z"}
{"levelname": "DEBUG", "module": "deploy_content", "funcName": "send_content", "message": "working dashboard", "dashboard": "deployfolder/Shared/Clinical/Clinical Effectiveness/Dashboard_9_CMS OP-22.json", "timestamp": "2022-09-22T17:48:51.110836Z"}
Traceback (most recent call last):
File "/usr/local/bin/ldeploy", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/looker_deployer/cli.py", line 46, in main
args.func(args)
File "/usr/local/lib/python3.8/site-packages/looker_deployer/commands/deploy_content.py", line 306, in main
send_content(
File "/usr/local/lib/python3.8/site-packages/looker_deployer/commands/deploy_content.py", line 268, in send_content
deploy_content("dashboard", dash, sdk, env, ini, debug)
File "/usr/local/lib/python3.8/site-packages/looker_deployer/commands/deploy_content.py", line 215, in deploy_content
a, b, c = dirs.partition(target_base)
TypeError: must be str, not NoneType

Looker Query ID API Patch

Looker recently patched the platform to Disallow Numeric Query Ids. Since this was rolled out errors show when running ldeploy content export. Temporarily toggling off this enabled ldeploy to function as expected.

@drstrangelooker

Deploy LookML to projects via advanced deploy mode

Suggested enhancement to deploy lookml to projects via adv deploy mode.

cd looker_deployer does not work/is not available in home directory

I have installed looker-deployer multiple times with success and get a response from "ldeploy --version". I attempted to use the looker-deployer to migrate content and was not able to specify an environment or a folder to migrate to. This is not going to work for us.

Looker-deployer has worked excellent in the past for us, but I am not stuck with it not working today, because we must stand up a new VM. The new VM install has not worked for me, yet.

Dashboard import error

When attempting to import a dashboard in local machine to a cloud Looker instance, the following error is displayed:

/Library/Ruby/Gems/2.6.0/gems/gazer-0.2.56/lib/gzr/commands/dashboard/import.rb:97:in 'block (3 levels) in execute': undefined method 'first' for nil:NilClass (NoMethodError)

This happens specifically in step {"levelname": "INFO", "module": "deploy_content", "funcName": "import_content", "message": "Deploying content", "content_type": "dashboard", ...

It seems that is coming from the gazer library, any ideas what might be the issue here?

Error on content import when creating destination folder

Hi @drstrangelooker @AdamMinton ,

I'm getting the same issue as @jtorreio.

However, it is happening when I run ldeploy content import. I get a generic looker_sdk error. As suggested, I tried gem update gazer and this did not resolve the issue.

Here is a screenshot of the error:

I've updated looker_deployer and looker_sdk and it hasn't done the trick either.

This is a high priority issue as we are blocked from deploying any Looker content to our instances. Can you please investigate?

Please let me know if you need anything from me. Thank you!

Originally posted by @jacobakh in #148 (comment)

Question: copying dashboard folders

Hi

Newbie here :-)
I would like to duplicate a whole Looker folder containing user-defined dashboards.
Within the same instance, and ideally also updating the model tiles derive from, as part of the same process.

Is this something looker_deployer is designed to support in some form?

Thanks in advance..

Looker API session times out after an hour

Hi everyone

I've got ldeploy content export working locally, but I'm trying to export content from a large instance, which takes a long time.

By default, the Looker API sessions on the SDK only last one hour, after which point I hit an authentication error.

Is there any way to configure the set up so that it automatically reauthenticates?

API Update Errors?

Hi @drstrangelooker, we're suddenly receiving failures of ldeploy with an error mentioning missing_method query_space. Full log below. Wondering if this is related to GA of API 4.0. I see in our older logs the connection string built ended in 3.1. Is there something we can switch on the frontend to force our string back to referencing API 3.1?

Full error (--- substituted for some detail)
[2022-03-16T16:11:51.581Z] /var/lib/gems/2.5.0/gems/looker-sdk-0.0.7/lib/looker-sdk/client/dynamic.rb:105:in `method_missing': undefined method `space' for #<LookerSDK::Client:0x000055b8a6c4dbc8> (NoMethodError) [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/gazer-0.2.44/lib/gzr/modules/space.rb:65:in `query_space' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/gazer-0.2.44/lib/gzr/commands/space/export.rb:87:in `process_space' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/gazer-0.2.44/lib/gzr/commands/space/export.rb:81:in `block in execute' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/gazer-0.2.44/lib/gzr/modules/session.rb:207:in `with_session' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/gazer-0.2.44/lib/gzr/commands/space/export.rb:49:in `execute' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/gazer-0.2.44/lib/gzr/commands/space.rb:80:in `export' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:115:in `invoke' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:238:in `block in subcommand' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start' [2022-03-16T16:11:51.581Z] from /var/lib/gems/2.5.0/gems/gazer-0.2.44/exe/gzr:36:in `<top (required)>' [2022-03-16T16:11:51.581Z] from /usr/local/bin/gzr:23:in `load' [2022-03-16T16:11:51.581Z] from /usr/local/bin/gzr:23:in `<main>' [2022-03-16T16:11:51.581Z] options: {"debug"=>true, "host"=>"---", "port"=>"443", "ssl"=>true, "verify_ssl"=>true, "timeout"=>60, "force"=>false, "client_id"=>"---", "client_secret"=>"---", "dir"=>"/looker_export/Shared"} [2022-03-16T16:11:51.581Z] API 3.1 available? true [2022-03-16T16:11:51.581Z] connecting to ["api_endpoint=>https://---:443/api/4.0", "connection_options=>{:ssl=>{:verify=>true, :verify_mode=>3}, :request=>{:timeout=>60}}", "user_agent=>Gazer 0.2.44", "client_id=>---", "client_secret=>---"] [2022-03-16T16:11:51.581Z] check for connectivity: true [2022-03-16T16:11:51.581Z] verify authentication: true [2022-03-16T16:11:51.581Z] logout

SSL Certificates could not be verified

@drstrangelooker - could you please take a look into this issue?

Here is my Dockerfile

FROM python:3.9

Install dependencies

RUN apt-get update -y && apt-get install -y ca-certificates

included all certs

RUN update-ca-certificates

ENV CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

add if using node

ENV NODE_EXTRA_CA_CERTS=${CURL_CA_BUNDLE}

add if using ruby

ENV BUNDLE_SSL_CA_CERT=${CURL_CA_BUNDLE}

add if using perl

ENV HTTPS_CA_FILE=${CURL_CA_BUNDLE}
ENV PERL_LWP_SSL_CA_FILE=${CURL_CA_BUNDLE}

add if using python

ENV REQUESTS_CA_BUNDLE=${CURL_CA_BUNDLE}
ENV SSL_CERT_FILE=${CURL_CA_BUNDLE}

RUN apt update
RUN apt -y install ruby ruby-dev
RUN gem install gazer

RUN apt -y install git
RUN git clone https://github.com/looker-open-source/looker_deployer.git

WORKDIR /looker_deployer

COPY looker.ini .
RUN pip install .

ENTRYPOINT ["ldeploy"]

docker run \
-v /Users/abc/repos/looker/ldeploy_settings:/ldeploy_settings
-v /Users/abc/repos/looker/ldeploy_output:/ldeploy_output
ldeploy content export
--debug
--ini /ldeploy_settings/looker.ini
--local-target /ldeploy_output
--env prod --folders 231
{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "main", "message": "ini file", "ini": "/ldeploy_settings/looker.ini", "timestamp": "2023-06-23T04:44:06.165962Z"}
{"levelname": "INFO", "module": "deploy_content_export", "funcName": "main", "message": "Exporting content", "env": "prod", "folders": ["231"], "dashboards": null, "looks": null, "local_target": "/ldeploy_output", "timestamp": "2023-06-23T04:44:06.166097Z"}
{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "recurse_folders", "message": "recursive folder crawl status", "current_id": "231", "folder_name": "myuser", "current_list": ["myuser"], "timestamp": "2023-06-23T04:44:07.031501Z"}
{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "recurse_folders", "message": "going for recursion", "parent_id": "2", "timestamp": "2023-06-23T04:44:07.031599Z"}
{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "recurse_folders", "message": "recursive folder crawl status", "current_id": "2", "folder_name": "Users", "current_list": ["myuser", "Users"], "timestamp": "2023-06-23T04:44:07.494477Z"}
{"levelname": "DEBUG", "module": "deploy_content_export", "funcName": "send_export", "message": "folder_list", "folder_id": "231", "list": ["Users", "myuser"], "timestamp": "2023-06-23T04:44:07.494613Z"}
options: {"debug"=>true, "host"=>"mylooker.cloud.looker.com", "port"=>"443", "ssl"=>true, "verify_ssl"=>false, "timeout"=>60, "force"=>false, "persistent"=>false, "token_file"=>false, "client_id"=>"myclientid", "client_secret"=>"mysecret", "dir"=>"/ldeploy_output/Users"}
using options ["debug=>true", "host=>mylooker.cloud.looker.com", "port=>443", "ssl=>true", "verify_ssl=>false", "timeout=>60", "force=>false", "persistent=>false", "token_file=>false", "client_id=>myclientid", "dir=>/ldeploy_output/Users"]
logout
ERROR: SSL Certificate could not be verified
Do you need the --no-verify-ssl option or the --no-ssl option?

ERROR: Dashboard already exists in space 739

We are using the below Ldeploy script for migrating our Dashboards from one environment to another which was working fine previously and is able to update the dashboard in the destination. but now it is giving the below error.

Error:
ERROR: Dashboard already exists in space 739

Command:
ldeploy content import --env destinationEnv --ini lookerEnvFile --debug --recursive --folders /sourceFolderPath --target-folder destinationFolderPath

send_boards function in ldeploy boards not finding existing dashboards in one client instance, but works on other instances

Typing the following in CLT:
(looker_deployment) brigitte % ldeploy boards --source dev --target clientname --allow-partial --board INPATIENT
Gets:
{"levelname": "WARNING", "module": "deploy_boards", "funcName": "send_boards", "message": "Missing content warning.", "missing_dashboards": [{"dash_id": 279, "dash_title": "Achieving Healthcare Potential for All"}, {"dash_id": 220, "dash_title": "Impact Analysis for QBRs"},

^ Yet both of these dashboards exist, and this works on other clients. I've tried removing and redploying the dashboards that aren't working, but got no luck.

Any ideas what could be happening here, @JCPistell ? It seems like the slug for a specific dashboard is not getting picked up at one client only.
Thx in advance for your help!
Brigitte