vancouver-datajam / gis-interpolation

Hackathon project for Datajam 2023 that will focus on interpolation of spatial data and displaying the data through a plotly dashboard.

Visualization & Prediction of Geospatial BC Climates

Project team members

• Team lead: Evie Lapalme
• Mentor: Sasha
• Documentation: Art, Ashley, Emilia
• Code Writing: Chloe, Caesar, Thomas, Art, Kelvin, Cecilia, Ashley, Melissa
• Code Review: Chloe, Caesar, Thomas, Art, Kelvin, Ashley, Melissa, Yiquan, Cecilia

Project statement

The aim of this project is to harness the power of data analysis and geospatial techniques to provide accurate visualizations and predictions of climatic conditions across various regions of British Columbia (BC). By employing Kriging interpolation on a carefully curated point dataset, we seek to create a detailed representation of BC's climate patterns. Additionally, we will develop an interactive geospatial dashboard using Plotly, enabling users to explore and understand the climatic variations in an intuitive and informative manner.

Presentation

https://docs.google.com/presentation/d/1uANv49etGWTlsa_tTqWVVDsW2Qh0-beMJwqWW1JWYPg/edit#slide=id.g226d46bec69_0_27

Key Tasks

• Kriging Interpolation
• Geospatial Dashboard Development
• Visualization and Interpretation
• Prediction and Analysis

Deliverables

• A well-documented Python codebase that encompasses the entire data analysis process, from data acquisition to dashboard development.
• An interactive geospatial dashboard powered by Plotly, providing easy access to climate information for different regions of BC.

What we accomplished

Kriging Interpolation:

• Implement the Kriging interpolation method to estimate climate values at unobserved locations.

Model Evaluation:

• The range of our data is -16.8 to 5.5 Celsius. Given that standard deviation is 5.33 and our errors are below this, this suggests that the model’s predictions are indeed closer to the actual values than the spread of the data!
• an R² score of 0.858 means that approximately 85.8% of the variability in the target variable (temperature in this case) can be explained by the model.

Interactive Dashboard:

• Present the interpolated climate data through visually appealing and informative plots and maps. Provide insights and context for interpreting the climatic patterns observed.
• Incorporating visual elements like maps, charts, and filters for a user-friendly experience.
• Enable users to input coordinates or select locations on the dashboard to receive predicted climate values. Offer insights into how the climate varies across different regions of British Columbia.

Why is this significant

Risk management and mitigation planning

• Geospatial interpolation is more important now than ever with climate change events worsening. This year we saw events like the fires in Hawaii due to poor disaster prevention and management - Moody’s RMS (Risk Management Solutions company) estimated up to $6 billion in economic losses from the devastating wildfires in Hawaii, which killed at least 115 people and destroyed countless homes and businesses. Geospatial interpolation would play a huge role in risk management and mitigation planning: By providing detailed spatial information on climate variables such as temperature (and more) geospatial interpolation supports risk assessment for extreme weather events, sea-level rise, and other climate-related hazards. This information is vital for designing mitigation and adaptation strategies, infrastructure planning, and disaster management. It could save lives and save billions in economic losses.

Skills

• Geospatial Data Analysis Techniques: Participants will gain hands-on experience in working with geospatial data, including data preprocessing, Kriging interpolation, and mapping.
• Interactive Dashboard Development: Participants will learn how to create dynamic and interactive geospatial dashboards using Plotly, enhancing their data visualization skills.
• Spatial Data Interpretation: This project will provide insights into how to interpret and visualize spatial data, allowing participants to understand climatic patterns at a regional level.
• Application of Geostatistical Methods: Participants will learn how to apply Kriging interpolation, a powerful geostatistical technique widely used in spatial analysis and prediction.
• Collaboration and Communication: Working as a team, participants will develop their skills in collaboration, communication, and sharing knowledge in a group setting.
• Project Documentation: Participants will gain experience in documenting their code, methodology, and results, which is essential for reproducibility and sharing findings.

Conclusion

This project presents an exciting opportunity to combine advanced data analysis techniques with geospatial visualization, offering a valuable resource for understanding and predicting climate patterns in British Columbia. Through this endeavor, we aim to contribute to a better-informed understanding of the climatic conditions in this diverse and ecologically significant region.

Description of Dataset

The dataset from https://climatebc.ca/ is a comprehensive collection of climate data specifically focused on the province of British Columbia (BC), Canada. It provides a wide range of climate information and analyses tailored to the unique geographical and environmental characteristics of BC. The dataset includes historical climate data, which can be crucial for understanding long-term climate trends, variability, and patterns in British Columbia. Additionally, it may contain model-based projections and forecasts, offering insights into future climate scenarios for the province.

Scope

• Temperature
• Precipitation
• Decade Averages from 2011-2021

Vancouver Datajam 2023 Schedule:

• Main page: https://vancouverdatajam.ca/

Important dates:

Date	Description
Sept 22	Projects are released, team introductions, familiarize with the Git
Sept 23	Main Hackathon Day
Sept 24	Wrap-up, judging, awards, and networking event

Time	Action item
Sept 22, 7pm	Team Meet & Greet
Sept 23, 9am	Team Stand-up
Sept 24, 9am	Team Stand-up
Sept 24, 11am	Deadline for Git repository finalization

gis-interpolation's Issues

Geospatial Dashboard Development

Tasks

• Library Installation: Install necessary Python libraries, including plotly, to support the creation of interactive dashboards. You can do this using pip:
  pip install plotly dash pandas geopandas
• Dashboard Layout Design: Plan and design the layout of the dashboard, including the arrangement of maps, charts, and filters.
• Data Integration: Integrate the Kriged climate surfaces and other relevant data into the dashboard for visualization.
• Interactive_ Elements: Implement interactive elements such as dropdown menus, sliders, and checkboxes to allow users to explore the data dynamically.
• Plotly Chart Creation: Use Plotly to create various types of charts (e.g., scatter plots, heatmaps) to visualize the climate data.
• Geospatial Mapping: Incorporate geospatial maps into the dashboard using Plotly's mapping capabilities to display the Kriged results.
• Data Filtering and Selection: Enable users to filter and select specific regions or timeframes within the dashboard to view targeted climate information.
• Dashboard Styling and Aesthetics: Customize the appearance of the dashboard, including colors, fonts, and overall design, for a polished and user-friendly interface.
• Testing and Debugging: Thoroughly test the dashboard functionality to ensure smooth interaction and address any potential issues.
• Documentation and User Guide: Provide clear documentation and user instructions for navigating and utilizing the interactive dashboard effectively.
• Integration with Kriging Results: Link the dashboard to the Kriged climate surfaces, allowing users to visualize both observed and interpolated climate data.

Resources

• Dashboard example: https://viz.pmel.noaa.gov/osmc/?color_by=platform_type
• Article: https://medium.com/plotly/building-a-big-data-geographical-dashboard-with-open-source-tools-c5108d7d5683

Step-By-Step Guide

Guide on creating a geospatial dashboard in Python using Plotly:

# Step 4: Import necessary libraries
import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output
import plotly.express as px
import geopandas as gpd

# Step 5: Create a Dash app
app = dash.Dash(__name__)

# Step 6: Load your geospatial data
gdf = gpd.read_file('path_to_your_shapefile.shp')

# Step 7: Define the layout of your dashboard
app.layout = html.Div([
    html.H1("Geospatial Dashboard"),
    
    # Dropdown for selecting a variable to visualize
    dcc.Dropdown(
        id='variable-dropdown',
        options=[
            {'label': column, 'value': column} for column in gdf.columns
        ],
        value=gdf.columns[0],  # Default selection
        multi=False
    ),
    
    # Geospatial map
    dcc.Graph(
        id='geo-map'
    )
])

# Step 8: Create callback function to update the map
@app.callback(
    Output('geo-map', 'figure'),
    Input('variable-dropdown', 'value')
)
def update_map(selected_variable):
    fig = px.choropleth(
        gdf,
        geojson=gdf.geometry,
        locations=gdf.index,
        color=selected_variable,
        projection="mercator"
    )
    
    fig.update_geos(fitbounds="locations", visible=False)
    fig.update_layout(
        title=f"{selected_variable} Distribution",
        coloraxis_showscale=True
    )
    
    return fig

# Step 9: Run the app
if __name__ == '__main__':
    app.run_server(debug=True)

Here's what each step does:

1. Import the necessary libraries, including Plotly, Dash, Pandas, and Geopandas.
2. Create a Dash app.
3. Load your geospatial data using Geopandas (replace 'path_to_your_shapefile.shp' with your actual file path).
4. Define the layout of your dashboard with HTML and Dash components.
5. Create a callback function to update the geospatial map based on user input (variable selection).
6. Run the app.

Kriging Interpolation

Tasks

Geostatistical Analysis:

Implement Kriging interpolation to estimate climate values at unobserved locations.

• Select Suitable Library: Choose a geostatistical library (e.g., scipy, PyKrige) for Kriging interpolation.
• Load and Prepare Data: Load the climate dataset containing observed values of precipitation, temperature, latitude, and longitude.
• Spatial Data Exploration: Visualize the spatial distribution of observed data points on a map to understand their geographic spread.
• Interpolation Setup: Define the variogram model (e.g., spherical, exponential) and other parameters necessary for Kriging interpolation.
• Split Data: Divide the dataset into training and testing sets, reserving a portion for model validation.
• Perform Kriging: Apply the chosen library's Kriging function to estimate climate values at unobserved locations based on the observed data.

Model Validation:

Evaluate the accuracy and performance of the Kriging model.

• Cross-Validation: Implement cross-validation by comparing the Kriged values to the observed values in the testing set.
• Calculate Prediction Errors: Compute error metrics (e.g., Root Mean Squared Error, Mean Absolute Error) to quantify the accuracy of the Kriging predictions.
• Generate Validation Plots: Create visualizations (e.g., scatter plots, histograms) comparing predicted and observed values to assess model performance.

Generate Kriged Climate Surfaces:

Create continuous climate surfaces for precipitation and temperature across British Columbia.

• Define Grid for Interpolation: Establish a spatial grid covering the entire British Columbia region to generate the continuous surfaces.
• Apply Kriging to Grid: Utilize the trained Kriging model to estimate climate values at each point within the grid.
• Create Visualizations: Generate maps displaying the Kriged climate surfaces for precipitation and temperature.
• Export Results: Save the Kriged climate surfaces as spatial datasets (e.g., raster files) for further analysis and visualization.

Resources

• Article & tutorial: https://towardsdatascience.com/kriging-the-french-temperatures-f0389ca908dd
• What is Kriging: https://gisgeography.com/kriging-interpolation-prediction/
• Article: https://www.publichealth.columbia.edu/research/population-health-methods/kriging-interpolation
• Methods https://pygis.io/docs/e_interpolation.html#kriging
• Example code: https://geostat-framework.readthedocs.io/projects/pykrige/en/latest/examples/04_krige_geometric.html
• ML in Python: https://pygis.io/docs/f_rs_ml_predict.html

Housekeeping

Tasks

• Project Relevance Check: Review the project objectives and ensure that the analysis and predictions align with the challenge of visualizing and predicting geospatial BC climates.
• Practicality Assessment: Evaluate the feasibility and practicality of the project's solutions in addressing the climate change challenge specific to British Columbia.
• Impact Evaluation: Assess the potential positive impact of the project's insights and predictions on climate-related decision-making and planning in British Columbia.
• Technical Complexity Review: Analyze the technical intricacy of the implemented Kriging interpolation, considering innovation and scalability in the context of geospatial climate analysis.
• Creativity Check: Evaluate the project's creativity in problem-solving, considering the uniqueness and originality of approaches taken in visualizing and predicting BC climates.
• Presentation Quality Assessment: Review the clarity, organization, and engagement level of the project's presentation, ensuring that information is effectively conveyed with appropriate use of visual aids.
• GitHub Repository Management: Confirm that the project's GitHub repository has a 'main' branch with all code changes merged, and that the README file is complete with comprehensive project documentation.
• Code Review and Documentation: Conduct a thorough review of the codebase to ensure readability, proper comments, and adherence to best practices, while also verifying that the documentation provides clear insights into the methodology.
• Dashboard User Experience (UX) Testing: Test the interactive dashboard's functionality and user interface to ensure a smooth and intuitive experience for users exploring climate data.
• Final Presentation Rehearsal: Conduct a final run-through of the presentation to practice delivery, address any potential hiccups, and ensure a confident and engaging presentation.

Submission

Requirements

• Readme
• In-Person Presentation: To be eligible for judging and prizes, the team must be present in person to present your project.

Judging Criteria

• Relevance & Practicality: Addresses the challenge and offers a relevant solution to tackle the problem.
• Potential Impact: Considers the significance of the climate change challenge addressed and positive change potential.
• Technical Complexity: Evaluates technical skills, innovation, and scalability potential.
• Creativity: Assesses uniqueness, innovative problem-solving, and originality.
• Presentation: Clarity of information conveyed, organization, visual aids usage, and engagement.