classicvalues / watson-discovery-analyze-data-breaches

A Node.js application that demonstrates how to Import, Enrich, and see Insights about data using Watson Discovery.

Use the Watson Discovery Service to analyze cyber security breaches

In this code pattern you will upload your own data into the Watson Discovery Service. Then you'll configure a web application so that it can query the data collection you created. The web app allows you to explore that data.

Once you have completed this code pattern, you will know how to:

• Build and run a Node.js API server with a HTML frontend written in React
• Configure Watson Discovery to build and enrich private data collections
• Use Watson Discovery to query and analyze data

Flow

1. The cyber breach json files are added to the Discovery collection.
2. The user interacts with the backend server via the app UI. The frontend app UI uses React to render search results and can reuse all of the views that are used by the backend for server side rendering. The frontend is using watson-react-components and is responsive.
3. User input is processed and routed to the backend server, which is responsible for server side rendering of the views to be displayed on the browser. The backend server is written using express and uses express-react-views engine to render views written using React.
4. The backend server sends user requests to the Watson Discovery Service. It acts as a proxy server, forwarding queries from the frontend to the Watson Discovery Service API while keeping sensitive API keys concealed from the user.

Included components

• Watson Discovery: A cognitive search and content analytics engine for applications to identify patterns, trends, and actionable insights.

Featured technologies

• Node.js - An asynchronous event driven JavaScript runtime, designed to build scalable applications
• React - Javascript library for building User Interfaces
• Express - A popular and minimalistic web framework for creating API and Web server

Watch the Video

Steps

Use the Deploy to IBM Cloud button OR create the services and run locally.

Deploy to IBM Cloud

1. Press the above Deploy to IBM Cloud button and then click on Deploy.

2. In Toolchains, click on Delivery Pipeline to watch while the app is deployed. Once deployed, the app can be viewed by clicking View app.

3. To see the app and services created and configured for this code pattern, use the IBM Cloud dashboard. The app is named watson-data-breaches with a unique suffix. The following services are created:

   • breaches-discovery-service

Run locally

NOTE: These steps are only needed when running locally instead of using the Deploy to IBM Cloud button.

1. Clone the repo
2. Create IBM Cloud services
3. Load the Discovery files
4. Configure credentials
5. Run the application

1. Clone the repo

Clone the watson-discovery-analyze-data-breaches repo locally. In a terminal, run:

git clone https://github.com/ibm/watson-discovery-analyze-data-breaches

We'll be using the folder data/breaches/

2. Create IBM Cloud services

Create the following services:

• Watson Discovery

3. Load the Discovery files

Launch the Watson Discovery tool. Create a new data collection and give the data collection a unique name.

Save the environment_id and collection_id for your .env file in the next step.

Under Add data to this collection use Drag and drop your documents here or browse from computer to seed the content with the json files in data/breaches/.

4. Configure credentials

The credentials for the IBM Cloud Discovery service can be found in the Services menu in IBM Cloud, by selecting the Service Credentials option for the service.

The other settings for Discovery were collected during the earlier setup steps (DISCOVERY_COLLECTION_ID and DISCOVERY_ENVIRONMENT_ID).

Copy the env.example to .env.

cp env.example .env

Edit the .env file with the necessary settings.

env.example

# Copy this file to .env and replace the credentials with
# your own before starting the app.

# Watson Discovery
DISCOVERY_URL=<add_discovery_url>
DISCOVERY_ENVIRONMENT_ID=<add_discovery_environment_id>
DISCOVERY_COLLECTION_ID=<add_discovery_collection_id>
## Un-comment and use either username+password or IAM apikey.
# DISCOVERY_USERNAME=<add_discovery_username>
# DISCOVERY_PASSWORD=<add_discovery_password>
# DISCOVERY_IAM_APIKEY=<add_discovery_iam_apikey>

# Run locally on a non-default port (default is 3000)
# PORT=3000

5. Run the application

1. Install Node.js runtime or NPM.
2. Start the app by running npm install, followed by npm start.
3. Use the chatbot at localhost:3000.

Note: server host can be changed as required in server.js and PORT can be set in .env.

Sample output

Troubleshooting

• Error: Environment {GUID} is still not active, retry once status is active

  This is common during the first run. The app tries to start before the Discovery environment is fully created. Allow a minute or two to pass. The environment should be usable on restart. If you used Deploy to IBM Cloud button the restart should be automatic.

• Error: Only one free environent is allowed per organization

 > To work with a free trial, a small free Discovery environment is created. If you already have a Discovery environment, this will fail. If you are not using Discovery, check for an old service thay you may want to delete. Otherwise use the .env DISCOVERY_ENVIRONMENT_ID to tell the app which environment you want it to use. A collection will be created in this environment using the default configuration.

Links

• Demo on Youtube: Watch the video.
• Watson Node.js SDK: Download the Watson Node SDK.
• Blog: Bring your own data to Watson Discovery Service: Steps to building the data for this code pattern.
• Data set: World’s biggest data breaches (Information is Beautiful): Check out the data set used for this code pattern, as well as others available for you to use.
• Watson Discovery, an IBM API adding value to corporate data: Dive into IBM Watson Discovery Service, enabling companies to structure and understand large masses of data.
• Blog: Watson Discovery Service – Understand your data at scale with less effort: Align and connect different data sets to expose critical correlations and causal factors.
• Blog: Using IBM Watson Discovery to query unstructured data: Make sense of and identify patterns in large amounts of unstructured data.

Learn more

• Artificial Intelligence Code Patterns: Enjoyed this code pattern? Check out our other AI Code Patterns.
• AI and Data Code Pattern Playlist: Bookmark our playlist with all of our code pattern videos
• With Watson: Want to take your Watson app to the next level? Looking to utilize Watson Brand assets? Join the With Watson program to leverage exclusive brand, marketing, and tech resources to amplify and accelerate your Watson embedded commercial solution.

License

This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache License, Version 2.

Apache License (ASL) FAQ