alexandermichels / cybergis-compute-python-sdk

CyberGIS-Compute Python SDK

CyberGIS-Compute is a scalable middleware framework for enabling high-performance and data-intensive geospatial research and education on CyberGISX. This API can be used to send supported jobs to various supported HPC & computing resources.

---

Installation

Requirements

• Python3 + pip3
• Jupyter server (Hub/Lab) with fixed domain
• System environment variables:
  • JUPYTERHUB_API_TOKEN: user access token, comes with JupyterHub/Lab.
  • JUPYTER_INSTANCE_URL: server url

From Pip:

pip install cybergis-compute-client

From Github:

git clone https://github.com/cybergis/cybergis-compute-python-sdk.git
cd cybergis-compute-python-sdk
python3 setup.py install

---

Hello World Example

In this example, you will be using the SDK's Pilot UI to run the hello world GitHub project on the Keeling Computing Cluster.

1. Run the Pilot UI

from cybergis_compute_client import CyberGISCompute

cybergis = CyberGISCompute(url="xxx") # replace xxx with CyberGIS-Compute server url
cybergis.show_ui() # run Pilot UI

2. Select hello world from 📦 Job Template
3. Select keeling_community from 🖥 Computing Resource
4. Configure the following, or leave it as default
   • Slurm Computing Configurations
   • Input Parameters
   • Receive Email
5. Select a file to upload under Upload Data
6. Click Submit

❓ If you wonder where the customized configuration options comes from, they are defined in the manifest.json file of each project. Please refer to https://github.com/cybergis/cybergis-compute-hello-world/blob/main/manifest.json

---

SDK Usage

cybergis = CyberGISCompute(url="xxx")

1. Query and resume jobs that you own.

# CyberGISCompute.list_job -> return a list of jobs that you submitted
cybergis.list_job()

# CyberGISCompute.get_job_by_id -> return a Job object referred by that id
cybergis.get_job_by_id(id)

2. Query CyberGIS-Compute server support information

# CyberGISCompute.list_hpc -> prints a list of hpc resources that the server supports
cybergis.list_hpc()

# CyberGISCompute.list_git -> prints a list of Git projects that the server supports
cybergis.list_git()

3. Use Pilot UI

# Renders a IPython Widget UI in Jupyter (async)
# CyberGISCompute.create_job_by_ui -> return None
cybergis.show_ui()

# Get the job submitted by the UI (after you press the submit button)
# CyberGISCompute.get_latest_created_job -> return Job object
cybergis.get_latest_created_job()

4. Submit job using programming style (in progress)

# the Job object is an interface for a job
# CyberGISCompute.create_job -> return Job object & print the job's information
job = cybergis.create_job(hpc="some HPC")

# Job.submit -> print the job's information
job.submit()

---

Related Documentation and Projects

• CyberGIS-Compute Python SDK webpage
  • Using CyberGIS-Compute
  • Hello World Notebook
  • Model Gallery
  • Contributing Models
• CyberGIS Compute Core - the server
• CyberGIS Compute Example Hello World Project