GithubHelp home page GithubHelp logo

esri / crowdsource-reporter-scripts Goto Github PK

View Code? Open in Web Editor NEW
13.0 32.0 11.0 1004 KB

Scripts that can be used to extend the functionality of crowdsource reporter

License: Apache License 2.0

Python 99.83% HTML 0.17%
data-management solutions local-government crowdsource-reporter python government crowdsourcing

crowdsource-reporter-scripts's Introduction

crowdsource-reporter-scripts

This repository contains a series of tools that can be used to extend the functionality of editable feature layers published to an ArcGIS Online organization or an ArcGIS portal. The scripts can be scheduled using an application such as Windows Tasks to scan the feature layers for new content.

Generate Report IDs

A script to generate identifiers for features using a defined sequence and a value that increments on a defined interval.

Requirements

Python 2.7, ArcPy

Configuration

For more information on configuring this script, see the documentation: Generate Report IDs

Download a supported version of the Service Functions script here, which contains the Generate Report IDs script.

Moderate Reports

A script to filter features based on lists of explicit and sensitive words. The script updates a flag field when a word on these lists is detected so that the feature no longer meets the requirements of a filter on a layer in a web map.

Requirements

Python 2.7, ArcPy

Configuration

For more information on configuring this script, see the documentation: Moderate Reports

Download a supported version of the Service Functions script here, which contains the Moderate Reports script.

Send Email Notifications

Send emails to specific email addresses, or to addresses in fields in the data. Multiple messages can be configured for each layer based on attribute values. The script updates a flag field to indicate that each message has been sent.

Requirements

Python 3, ArcGIS API for Python

Configuration

For more information on configuring this script, see the documentation: Send Email Notifications

Download a supported version of the Service Functions script here, which contains the Send Email Notifications script.

Enrich Reports

Calculate feature attributes based on the attributes of co-incident features.

Requirements

Python 3, ArcGIS API for Python

Configuration

For more information on configuring this script, see the documentation: Enrich Reports

Download a supported version of the Service Functions script here, which contains the Enrich Reports script.

Connect to ArcGIS Workforce

Create workforce assignments from incoming Crowdsource Reporter, GeoForm, and Survey 123 reports.

Requirements

Python 3, ArcGIS API for Python

Configuration

For more information on configuring this script, see the documentation: Connect to ArcGIS Workforce

Download a supported version of the Connect to ArcGIS Workforce script here

Cityworks Connection (Developed in Partnership with Cityworks)

A script to pass data from editable ArcGIS feature layers to Cityworks tables, including related records and attachments. The script also passes the Cityworks Request ID and open date back to the ArcGIS feature.

The script assumes that the data is being collected using the Crowdsource Reporter application. For input, it requires the group containing the maps that are visible in the Crowdsource Reporter app.

Note: This integration requires specific versions of the Cityworks platform and integration with existing service request content. If you would like to integrate Citizen Problem Reporter with your Cityworks implementation, please reach out to your Cityworks account representative. They will be able to help you with specific system requirements and the steps required to complete the integration.

Requirements

ArcGIS Pro 2.2+ Python 3.5+, ArcGIS API for Python 1.4.1+

Configuration
  1. If not previously installed, use the Python Package Manager in ArcGIS Pro to install the ArcGIS API for Python (package name: arcgis)
  2. In ArcGIS Pro, install the Solutions Deployment Tool, and use it to deploy the Citizen Problem Reporter solution to your portal or organization. If necessary, use this tool to add fields to the layers and to update the domains to match the report types found in Cityworks.
  3. In ArcGIS Pro, add the Connect2Cityworks toolbox to your current project.
  4. Open the tool contained in the toolbox.
  5. Provide the connection information for both Cityworks and ArcGIS Online/ArcGIS Enterprise.
  6. Choose the group used to configure the Crowdsource Reporter application being used to collect the data.
  7. Choose the layers that the tool should process. These parameters will list all layers from the maps in the group that have at least one editable field configured in the popup.
  8. Choose the Cityworks field that contains the Request ID, and the ArcGIS field where this value should be recorded. Only ArcGIS fields found in all selected layers will be options for the ArcGIS Report ID Field parameter.
  9. Choose the Cityworks field that contains the open date of the service request, and the ArcGIS field where this value should be recorded. Only ArcGIS fields found in all selected layers will be options for the ArcGIS Open Date Field parameter.
  10. Choose the Cityworks field and the ArcGIS field that contains the type of request generated by the report. Both fields must support identical values. Only ArcGIS fields found in all selected layers will be options for the ArcGIS Report Type Field parameter.
  11. Choose the field used to indicate the status of the report with regards to being transferred to Cityworks. Specify the value of this field when the report needs to be transferred in the Flag On Value parameter, and the value once the transfer is complete in the Flag Off Value parameter.
  12. Map other field pairs to transfer additional information received with the report to Cityworks. Some Cityworks fields you will want to consider are: CallerFirstName, CallerLastName, CallerAddress, CallerCity, CallerState, CallerZip, CallerHomePhone, CallerEmail, Details, Address. Map an unused Cityworks Universal Custom Field to the ArcGIS field OBJECTID. For example Num5 to OBJECTID.
  13. Optionally, choose editable tables from the maps to process as well as the layers. These related records will be copied to Cityworks as additional information on the report. As with the layers, map the ArcGIS and Cityworks fields to specify which data should be transferred. To transfer comments, map the following Cityworks fields: FirstName, LastName, Comments.
  14. Finally, choose a location to save out this information in a configuration file. This configuration file will be read by the script that will handle the transfer of information between ArcGIS and Cityworks.

The following is an example of what a config file might look like:

{
    "cityworks": {
        "username": "apiuser",
        "password": "apipassword",
        "url": "https://cityworks.your-org.com/CityworksSite",
        "timezone": "America/Denver",
        "isCWOL": false
    },
    "fields": {
        "ids": [
            "RequestId",
            "REPORTID"
        ],
        "type": [
            "ProblemSid",
            "PROBTYPE"
        ],
        "opendate": [
            "DateTimeInit",
            "submitdt"
        ],
        "layers": [
            [
                "CallerAddress",
                "ADDRESS"
            ],
            [
                "CallerCity",
                "CITY"
            ],
            [
                "CallerState",
                "STATE"
            ],
            [
                "CallerZip",
                "ZIP"
            ],
            [
                "CallerFirstName",
                "FNAME"
            ],
            [
                "CallerLastName",
                "LNAME"
            ],
            [
                "CallerHomePhone",
                "PHONE"
            ],
            [
                "CallerEmail",
                "EMAIL"
            ],
            [
                "Details",
                "DETAILS"
            ],
            [
                "Address",
                "LOCATION"
            ],
            [
                "Num5",
                "OBJECTID"
            ]
        ],
        "tables": [
            [
                "FirstName",
                "FNAME"
            ],
            [
                "LastName",
                "LNAME"
            ],
            [
                "Comments",
                "COMMENT"
            ]
        ]
    },
    "flag": {
        "field": "processed",
        "on": "No",
        "off": "Yes"
    },
    "arcgis": {
        "username": "ArcGISUser",
        "password": "ArcGISPassword",
        "layers": [
            "https://services.arcgis.com/5555555555555555/arcgis/rest/services/Citizen_Request_Portal/FeatureServer/2",
            "https://services.arcgis.com/5555555555555555/arcgis/rest/services/Citizen_Request_Portal/FeatureServer/1",
            "https://services.arcgis.com/5555555555555555/arcgis/rest/services/Citizen_Request_Portal/FeatureServer/0"
        ],
        "tables": [
            "https://services.arcgis.com/5555555555555555/arcgis/rest/services/Citizen_Request_Portal/FeatureServer/4",
            "https://services.arcgis.com/5555555555555555/arcgis/rest/services/Citizen_Request_Portal/FeatureServer/3",
            "https://services.arcgis.com/5555555555555555/arcgis/rest/services/Citizen_Request_Portal/FeatureServer/5"
        ],
        "url": "https://yourorg.maps.arcgis.com"
    }
}

To execute the script that transfers data between ArcGIS and Cityworks, configure an application such as Windows Task Scheduler.

  1. Open Windows Task Scheduler
  2. Click Action > Create Task and provide a name for the task.
  3. Click the Action tab and click New.
  4. Set Action to Start a Program.
  5. Browse to the location of your Python 3 installation (for example, \Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-python3\python.exe).
  6. In the Add arguments text box, copy the name of the script (connect_to_cityworks.py) and the path to the configuration file save from running the tool in ArcGIS Pro.The script name and the configuration file path must be separated by a script, and the configuration file path must be surrounded with double quotes if it contains any spaces.
  7. In the Start in text box, type the path to the folder containing the scripts and email templates and click OK.
  8. Click the Trigger tab, click New, and set a schedule for your task.
  9. Click OK.

General Help

Resources

Learn more about Esri's ArcGIS for Local Government maps and apps.

Show me a list of other Local Government GitHub repositories.

Issues

Find a bug or want to request a new feature? Please let us know by submitting an issue.

Contributing

Esri welcomes contributions from anyone and everyone. Please see our guidelines for contributing.

Licensing

Copyright 2016 Esri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's LICENSE file.

crowdsource-reporter-scripts's People

Contributors

allisonmuise avatar aparker-cityworks avatar azizaparveen avatar chris-fox avatar cosbyr avatar lindsaythomas avatar miketschudi avatar mottesen-trimble avatar scottoppmann avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

crowdsource-reporter-scripts's Issues

issue with subbing values not from fields

There is an error in the script. Line that says:
email_subject = email_subject.replace(sub, str(feature.attributes[message['body substitutions'][sub]]))

This tries to replace using a field value, but you are building the URL with configured strings, not field values. It works if you change this line to say:
email_subject = email_subject.replace(sub, str(message['body substitutions'][sub]))

add message to cityworks tool

when problem type isn't found in cityworks, it could be either of 2 reasons

  • typo/problem doesn't exist as written
  • report type in CW doesn't have 'For Public Website' option enabled.

reduce connections to email server to 1/script run

Causes connection failures to AWS when too many connections are attempted too quickly.

script currently connects to email server for every service. Remove With statement that initiates email server connection and move the connection to outside the services loop. Reference the server connection inside the email processing section for each service. Close email server connection as part of Finally block.

sql parameters

Couldn't use SQL parameters in v1 due to pro bug. Try to re-implement SQL builder param

One of maps wouldn't open on a mobile

In our citizen reporter application one of the maps stopped working (wouldn't open) when using application on a mobile device. It works as expected on the desktop.

Here is our application: Omaha Hotline

The problem category is Graffiti

Thank you,
Nataliya

No module named 'send_email'

Hello,
I'm attempting to run servicefunctions.py from Jupyter Notebook, but it doesn't recognize the send_email module:
from send_email import EmailServer

ENH-000116673 Provide the ability to enter multiple email addresses in the Recipient Email Address section when using the Send Emails script found within the ServiceFunctions toolbox.

Salesforce ID: ENH-000116673
Salesforce Submitter: Matthew Metten
Salesforce Submit Date: 9/10/2018 2:37 AM
Salesforce Bug Type: Product
Salesforce Severity: Medium
Repro Data: (n/a)
Work Around: User Story: The user and his organization is utilizing this Solution/Send Email tool to send emails to a large number of people (20 to 40). The overall workflow would require the user to redundantly add the same information (Email Template, SQL Query, etc) 20 to 40 times just to include more than recipient. Providing/allowing this functionality would increase the efficiency of the tool and also allow for a more streamlined/modern tool for setting up such a function. 

Steps to Demonstrate: 
  1. Download and extract the ServiceFunctions toolbox.
  2. Open the toolbox in ArcGIS Pro 2.x and, if you've never used the toolbox before, run the Define Connection Settings tool to establish a connection between the toolbox and the ArcGIS Enterprise portal or ArcGIS Online organization containing the layers which will be used to trigger and generate the emails.
  3. Open the Send Emails tool.
  4. If the tool has not been run previously, start by specifying the connection information for your SMTP server in the General Email Settings section of the tool by providing the following values:
  • SMTP Server—URL of the SMTP server used for sending emails (within Esri network: SMTP.esri.com:25)
  • If outside the network:
  • SMTP Username—The username required to authenticate to the SMTP server. This is not required if authenticating through a port.
  • SMTP Password—The password required to authenticate to the SMTP server. This is not required if authenticating through a port.
  • From Address—The address from which the emails should be sent.
  • Reply Address—The address that should be used for any replies to the email message.
  1. Browse to the path of an HTML template for the body of the email (found in the servicesupport download folder).
  2. Provide a valid SQL Query (1=1)
  3. Provide the email address of the recipient and note that nowhere in the doc or the tool does it explain if you can or cannot add multiple recipients. Additionally, adding any delimiter and an additional email address does not provide the ability to send the email to multiple recipients.
  4. Provide a subject and a field to update and select Run.
  5. Navigate to the send_email Python script and run with ArcGIS Pro IDLE.
  6. Overall outcome will be one recipient even if multiple addresses are added. In the received email it shows the other addresses but these are never actually sent properly.

Product: ArcGIS for Local Government
Functional Category: ArcGIS for Local Government
Client Platform: (n/a)
Version Found: 2.2
Planned Version Fixed: (n/a)
Comment: (n/a)

issue selecting a table from the current map

Using either the Moderate or the Email tools:
When I select a table layer as the input layer (eg the comments layer from a Reporter config), I'm having a hard time getting the URL to that layer. The object that is returned is a geoprocessing value object that has no .describe, .properties, or .connectionProperties.

I can get to the URL if the map containing that layer is active by navigating through the current project and active map properties (thanks @cosbyr !). If the user navigates away from that map after selecting the table, I lose the URL. I also can't get the URL in the first place if that map isn't active, even though the table will show in the drop down.

suggestions?
@chris-fox @cosbyr @ChrisBuscaglia

Define Portal Settings enhancement to get Portal connections in Pro

What if instead of asking the user to input their Portal URL, you make it a pick list using URLs the user added to Pro. You can use arcpy.ListPortalURLs() to return all the Portal URLs the user currently has added to Pro. You can use GetActivePortalURL() to set the default for the parameter to the active portal. I think this would be a nice enhancement and get around the issue of the user typing in the url incorrectly. I think it is ok to restrict them to Portals defined in Pro because the Feature Layer params in other tools will fail to validate the service if it is secured and you don’t have a valid Portal connection to it.

Similarly for the username you can use arcpy.GetPortalDescription(URL)['user']['username'] to get the username for the portal connection and default that when they select a Portal URL. You need to check if the key ‘user’ exists in the returned dictionary because if they are not signed in this key won’t exist.

Recommend using %c for strftime of value subsitution

In the _get_value function dates are being formatted as '%d-%m-%Y %H:%M:%S'. If you use '%c' instead the date/time will be returned in the system locale’s appropriate date and time representation.

            try:
                val = dt.fromtimestamp(
                    row.attributes[sub]).strftime('%c')
            except OSError:  # timestamp in milliseconds
                val = dt.fromtimestamp(
                    row.attributes[sub] / 1000).strftime('%c')

Send Email parameter help

  • email spelled wrong in tool metadata tags
  • typo in 'Sent Values' in SQL Query description in email settings param
  • typo in 'sending' in Sent Value description in email settings param
  • I think the information about authenticating through a port is misleading in the smtp_server, username and password params. To send an e-mail through my gmail I need to provide the port and my gmail username and password. I think the fact that username and password are optional params is enough to say they are not required and it is unrelated to providing a port.
  • typo in Reply Address param help 'replies sto the'

BUG-000113756 The ServiceFunction.py script produces errors indicating it could not read the email template when no substitution is used in the email.

Salesforce ID: BUG-000113756
Salesforce Submitter: Andy Shoemaker
Salesforce Submit Date: 5/04/2018 1:35 AM
Salesforce Bug Type: Usability
Salesforce Severity: High
Repro Data: (n/a)
Work Around: Set a substitution in the SendEmails tool.
Product: ArcGIS Pro
Functional Category: ArcGIS for Local Government
Client Platform: (n/a)
Version Found: 2.1.2
Planned Version Fixed: (n/a)
Comment: (n/a)

Suggestion - Step 6- Type vairable

Type Variable - either we can specify this is not a datatype of the field (in addition to the description we have already) or change the variable name, so end users don't get confused (if they are skimming through the doc) as 'type' is preceded by 'field' variable.

Crowdsource to Cityworks integration error: json.decoder.JSONDecodeError

I couple months ago I updated my test Cityworks instance to the 15.6.1. Ever since I am not able to run the Crowdsource to Cityworks integration script. I am getting the following error:
image

The line 245 in my script is relate to the authentication:
image

And here is what I have in the script for establishing the connection:
image

I was wondering if something has changed in the Cityworks 15.6.1 and how authentication is done?

Thanks,
Nataliya

Connect2Cityworks.pyt KeyError operationalLayers

When I try to run the Connect2Cityworks tool in ArcGIS Pro, I get a KeyError on 'operationalLayers'. It looks like this line is causing the issue:
for layer in webmap['operationalLayers']:

Changing the line to the following appears to resolve the issue:
for layer in webmap.definition['operationalLayers']:

ENH-000114300 Allow apps that use crowd source reporter to moderate comments.

Salesforce ID: ENH-000114300
Salesforce Submitter: Katrina McLean
Salesforce Submit Date: 5/29/2018 10:57 AM
Salesforce Bug Type: Product
Salesforce Severity: Low
Repro Data: (n/a)
Work Around: User Story: It would be good to be able to moderate the comments so that any inappropriate comments can be removed. It would be best to have this adapted in the moderate reports stand alone script, but alternatively this would be a great option in ArcGIS Online from the solutions deployment tool.

Steps to Demonstrate:
Install the moderate reports toolbox and try to moderate comments in a related table. It will give an error.
Deploy Crowd Reporting application. There is no option to moderate incoming comments.

Product: ArcGIS for Local Government
Functional Category: ArcGIS for Local Government
Client Platform: (n/a)
Version Found: (n/a)
Planned Version Fixed: (n/a)
Comment: (n/a)

review instructions

@aparker-cityworks please review the following so I can move this tool into Master:

  • Review the instructions for this tool in the ReadMe
  • Review the in-tool parameter help
  • Verify that you guys are ok with the tool & toolbox names ( @ScottOppmann would like your ok here, too)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.