redislabs-training / ru204

This is the course repository for the Redis University RU204 Storing, Querying, and Indexing JSON at Speed course.

RU204: Storing, Querying and Indexing JSON at Speed

Introduction

This repository contains example data and setup instructions for the Redis University course RU204: Storing, Querying, and Indexing JSON at Speed.

To take this course, you'll first need to do the following:

1. Make sure that you have Python 3.7 or newer installed on your system (we've tested this with Python 3.9). Check your Python version with python --version.
2. Clone this git repository from GitHub.
3. Get a Redis Stack instance locally or in the cloud.
4. Install RedisInsight.
5. Load the course sample data and create the search index.

We've provided detailed instructions for each of these steps below.

Throughout the course, we've provided example code written in Python, JavaScript (Node.js), Java and C#. You don't need to run the code to be successful on this course, and the exam does not contain programming language specific questions. If you'd like to try running some or all of the code samples then you'll also need to install:

• Node.js (version 14.8 or higher)
• Java JDK (Java 11 or higher)
• .NET SDK (version 6 or higher)

Clone the Course Git Repository

There are two ways to get a copy of this git repository on your machine...

Option 1: Use git clone

If you have the git command line tools installed, open up a terminal instance and clone the repository like this:

git clone https://github.com/redislabs-training/ru204.git

Now, change to the directory containing the course files:

cd ru204

The remainder of these instructions assume that you start from a terminal session whose current directory is ru204.

Your next step is to proceed to "Redis Setup".

Option 2: Download a Zip File from GitHub

If you don't have the git command line tools, download the repository as a zip file from GitHub and unzip it on your local machine.

Now, open up a terminal instance and change to the directory containing the course files:

cd ru204

The remainder of these instructions assume that you start from a terminal session whose current directory is ru204.

Your next step is to proceed to "Redis Setup".

Redis Setup

This course requires an instance of Redis with the RediSearch and RedisJSON modules installed. These are part of the Redis Stack product. Review the following options and choose the one that's right for you.

Option 1: Redis Cloud

This option doesn't require you to install Redis on your local machine. We provide a free instance of Redis in the cloud that you can keep and use for your own projects once you're finished with the course.

Let's get up and running with Redis in the cloud:

• Use your browser to navigate to the signup page on redis.com.
• Complete the signup form and click the "Get Started" button. Note that you can sign up with your Google or GitHub account if you prefer.
• When you receive the activation email from Redis, open it and click on the "Activate Account" button.
• You'll be taken to the dashboard, and a New Subscription dialog appears:

• Select your preferred cloud provider and a region close to you. This is where we'll host your free Redis instance for you.
• Click the "Let's start free" button.
• Your new free database will be created, and you should see something like this:

• Click on "Redis-free-db" to drill down into the details for your instance.
• To set up RedisInsight and connect the course data loader script to your database, you'll need to gather the following items from the dashboard and keep a record of them:
  • Host name
  • Port
  • User name (this will be default)
  • Password
• Your host name and port can be found in the "General" section (see below for example, here the host name is redis-15676.c10.us-east-1-3.ec2.cloud.redislabs.com and the port is 15676).

• Scroll down to the "Security" section.
• Click the "Copy" button next to "Default user password" to copy the Redis password into the clipboard. Paste this somewhere for safekeeping. Alternatively, click the eye icon to show the password:

• If you see a green check mark next to the "Redis-free-db" title, your database is ready to use!

Your next step is to set up RedisInsight...

Option 2: Docker

This is the most straightforward option if you want to run Redis locally. First, make sure you have Docker installed.

This course uses the Redis Stack Docker container. Download and start it with Docker Compose as follows:

docker-compose up -d

Check that the container is running like this:

docker ps

You should see output that includes the following:

CONTAINER ID IMAGE                    COMMAND          CREATED           STATUS       PORTS                                          NAMES
f7b89c3ca0dd redis/redis-stack:latest "/entrypoint.sh" About an hour ago Up 8 seconds 0.0.0.0:6379->6379/tcp, 0.0.0.0:8001->8001/tcp redisu-ru204

Leave the container running for now. When you want to stop it, use this command:

docker-compose down

The container will persist Redis data to the redisdata folder.

Your next step is to set up RedisInsight...

Option 3: Install Redis Stack

Redis Stack can be installed using popular package managers for macOS and Linux. Follow the instructions on redis.io to install and start Redis Stack.

Your next step is to set up RedisInsight...

RedisInsight Setup

RedisInsight is a graphical interface allowing you to interact with data and configure Redis instances.

Option 1: Install and Configure the Desktop Application

The preferred way to run RedisInsight is as a desktop application. If you're using a Redis instance in the cloud, you'll need to install the RedisInsight desktop application. If you chose the Docker or local install of Redis Stack option for your Redis instance, you can either install the RedisInsight desktop application or choose run it as a web application with no further software installation required (see option 2 below).

To use the desktop application, first download and install it using the instructions on the RedisInsight download page.

Once you have installed RedisInsight, start it up and agree to the terms and conditions of use.

Finally, configure RedisInsight to connect to your Redis Instance... Click the "ADD REDIS DATABASE" button and fill out the new database form with the following values.

If your Redis instance is in the cloud:

• Host: The host name for your cloud database
• Port: The port number for your cloud database
• Database Alias: RU204
• Username: default
• Password: The password for your cloud database
• Select Logical Database: [leave unchecked]
• Use TLS: [leave unchecked]

If you are using Docker or you have installed Redis Stack locally:

• Host: localhost
• Port: 6379
• Database Alias: RU204
• Username: [leave blank]
• Password: [leave blank]
• Select Logical Database: [leave unchecked]
• Use TLS: [leave unchecked]

Then click the "Add Redis Database" button to connect to your Redis instance. You should see "RU204" in the list of Redis databases, and can click on it to open the Browser view.

If you see an error while trying to connect to Redis, ensure that your Redis instance is running (make sure the Docker container is up or your locally installed Redis Stack is running) and try again.

Your next step is to load the sample data...

Option 2: Use the Web Interface

If you used the Docker or local install of Redis Stack options to get your Redis instance, you can choose to use RedisInsight as a web application with no further software to install. If you're using a Redis instance in the cloud, this option is not currently available to you and you should download and install RedisInsight instead.

First, ensure that the Docker container or your local Redis Stack installation is running.

Now, point your browser at http://localhost:8001/ and you should see the RedisInsight terms and conditions:

Accept the terms and click "Submit". RedisInsight will automatically connect to the local Redis Stack instance and display the key browser:

Your next step is to load the sample data...

Load the Sample Data into Redis and Create the Search Index

This course uses a sample data set consisting of almost 1500 JSON documents, each containing details about a science fiction book. These documents are contained in the data/books folder.

Before starting the course, you'll need to use the data loader to load these documents into Redis. The data loader also sets up a RediSearch index that you'll use to search the document collection.

First, with your shell in the ru204 folder create a Python virtual environment, activate it and install the dependencies required to run the data loader:

python3 -m venv venv
. ./venv/bin/activate
pip install -r requirements.txt

Now run the data loader script, passing it the name of the folder containing the JSON data files to load into Redis.

If your Redis instance is in the cloud, start the data loader like this:

python data_loader.py --dir data/books --redis redis://default:password@host:port/

Be sure to replace password, host and port with the values for your cloud instance.

If you're using Docker, or have installed Redis Stack locally, start the data loader this way:

python data_loader.py --dir data/books

You should expect to see output similar to the following:

Connecting to Redis at redis://localhost:6379/
Deleting any existing JSON documents for RU204.
Dropping any existing search index.
Creating search index.
Loading JSON files from data/books.
Stored book In Real Life at key ru204:book:3721.
Stored book Lines of Departure at key ru204:book:729.
...
Lots more loading of books...
...
Stored book Good Omens: The Nice and Accurate Prophecies of Agnes Nutter, Witch at key ru204:book:28376.
Stored book Johannes Cabal and the Blustery Day: And Other Tales of the Necromancer at key ru204:book:35.
Stored book Eternity's Wheel at key ru204:book:182.
Stored book Shards and Ashes at key ru204:book:478.
Loaded 1486 books into Redis.
Data verification checks completed OK.

If you see the "Loaded 1486 books into Redis." and "Data verification checks completed OK." messages then you've completed the setup steps and are ready to go.

You're Ready!

You're now ready to take the course! If you haven't already, sign up here to access the course materials.

You're Not Alone!

If you need help or want to chat about all things Redis, join us on our Discord server where you'll find a dedicated channel #ru204-storing-querying-and-indexing-json-at-speed for this course.

ru204's Issues

Section 2 . NET code update `url` to `uri`

For consistency with section 4, update url to uri:

https://github.com/redislabs-training/ru204/blob/main/src/dotnet/redis_om_example/Program.cs

Check if this appears in any HTML and update that too.

Section 4 - .NET example code only connects to local Redis

In section 4, the .NET example code only connects to Redis on localhost:6379. Update the HTML document and source code to have a configurable Redis connection in the same way that section 2 does.

Error while loading data

I get this error when I am trying to load data in Redis:
python: 3.9.18 (anaconda)
Redis: 7.2.3
OS : Debian 12

(ru204) root@localhost:~/ru204#python data_loader.py --dir data/books --redis redis://shoobi:[email protected]:6379/
Connecting to Redis at redis://shoobi:[email protected]:6379/
Deleting any existing JSON documents for RU204.
Dropping any existing search index.
Creating search index.
Traceback (most recent call last):
File "/root/ru204/data_loader.py", line 41, in
r.ft(INDEX_NAME).create_index(
File "/opt/anaconda3/envs/ru204/lib/python3.9/site-packages/redis/commands/search/commands.py", line 139, in create_index
return self.execute_command(*args)
File "/opt/anaconda3/envs/ru204/lib/python3.9/site-packages/redis/client.py", line 1238, in execute_command
return conn.retry.call_with_retry(
File "/opt/anaconda3/envs/ru204/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/opt/anaconda3/envs/ru204/lib/python3.9/site-packages/redis/client.py", line 1239, in
lambda: self._send_command_parse_response(
File "/opt/anaconda3/envs/ru204/lib/python3.9/site-packages/redis/client.py", line 1215, in _send_command_parse_response
return self.parse_response(conn, command_name, **options)
File "/opt/anaconda3/envs/ru204/lib/python3.9/site-packages/redis/client.py", line 1254, in parse_response
response = connection.read_response()
File "/opt/anaconda3/envs/ru204/lib/python3.9/site-packages/redis/connection.py", line 839, in read_response
raise response
redis.exceptions.ResponseError: Could not parse field spec

Add transcripts for videos when we have them

Add the .srt transcript files for each video to this repo when we have them. Store them in courseware/transcripts using the same folder layout and naming strtucture as the existing html folder does.

Add a README in the `transcripts` folder

Add a README in the transcripts folder so that students know what to expect there and what the format of the files is.

LInk to command's document page the first time a command is introduced.

Fix section 1 image names to have full section numbers that match the corresponding HTML documents.

Affected image files:

• All 4 png files in courseware/images/section_1.

Section 5.2 check wording around `FT.ALTER`

Raised on Discord:

The FT.ALTER command is used to add additional attributes to an existing index. This triggers a reindexing of all documents with the new attribute. But at the end, It should be noted that FT.ALTER does not trigger a reindex of previously indexed documents with the updated attributes.

Check into this (reference page is https://redis.io/commands/ft.alter/)

Typo in `NUMINCRBY` exercise in 1.4.1

Discord user "" found a typo in 1.4.1:

Hi all,
just to mention a typo...
Course Section 1 - Managing JSON Data in Redis 1.4 - Updating Data with RedisJSON Hands-On Exercise
The JSON.NUMINCRBY command may also be used to decrement numerical values as well. Using a negative number performs subtraction upon any numerical value within the document. Let's reduce the rating_votes by 10:

JSON.NUMINCRBY ru204:book425 $.metrics.rating_votes 23
Redis responds with the updated value for rating_votes:
[3426]

JSON.NUMINCRBY ru204:book425 $.metrics.rating_votes 23 should be
JSON.NUMINCRBY ru204:book425 $.metrics.rating_votes -10

Fix this!

Aftertime is needed in 3.2, deleted at end of 1.4

In the course of learning JSON.DEL we delete the book ru204:book:425, this causes problems in Section 3 as we need the book as a search example.

Add Section 6 (Final Exam) HTML files to this repository.

Add the HTML files for section 6.

Add alt text to all images in the courseware.

Add alt text to all images in the courseware.

data_loader.py is not working on Mac Arm M1

I followed required steps on Mac Arm M1 and unable to get data_loader.py script working. Getting below errors in virtual environment.

python3 data_loader.py --dir data/books
Connecting to Redis at redis://localhost:6379
Deleting any existing JSON documents for RU204.
Dropping any existing search index.
Creating search index.
Traceback (most recent call last):
File "/Users/akanse/redis-course/ru204/data_loader.py", line 42, in
r.ft(INDEX_NAME).create_index(
File "/Users/akanse/redis-course/ru204/venv/lib/python3.9/site-packages/redis/commands/search/commands.py", line 139, in create_index
return self.execute_command(*args)
File "/Users/akanse/redis-course/ru204/venv/lib/python3.9/site-packages/redis/client.py", line 1238, in execute_command
return conn.retry.call_with_retry(
File "/Users/akanse/redis-course/ru204/venv/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/Users/akanse/redis-course/ru204/venv/lib/python3.9/site-packages/redis/client.py", line 1239, in
lambda: self._send_command_parse_response(
File "/Users/akanse/redis-course/ru204/venv/lib/python3.9/site-packages/redis/client.py", line 1215, in _send_command_parse_response
return self.parse_response(conn, command_name, **options)
File "/Users/akanse/redis-course/ru204/venv/lib/python3.9/site-packages/redis/client.py", line 1254, in parse_response
response = connection.read_response()
File "/Users/akanse/redis-course/ru204/venv/lib/python3.9/site-packages/redis/connection.py", line 839, in read_response
raise response
redis.exceptions.ResponseError: unknown command 'FT.CREATE', with args beginning with: 'idx:books' 'ON' 'JSON' 'PREFIX' '1' 'ru204:book:' 'SCORE' '1.0' 'SCHEMA' '$.author' 'AS' 'author' 'TEXT' 'WEIGHT' '1.0' 'SORTABL'

Note that required modules successfully installed -
pip install -r requirements.txt
Requirement already satisfied: async-timeout==4.0.2 in ./venv/lib/python3.9/site-packages (from -r requirements.txt (line 1)) (4.0.2)
Requirement already satisfied: Deprecated==1.2.13 in ./venv/lib/python3.9/site-packages (from -r requirements.txt (line 2)) (1.2.13)
Requirement already satisfied: packaging==21.3 in ./venv/lib/python3.9/site-packages (from -r requirements.txt (line 3)) (21.3)
Requirement already satisfied: pyparsing==3.0.9 in ./venv/lib/python3.9/site-packages (from -r requirements.txt (line 4)) (3.0.9)
Requirement already satisfied: redis==4.3.4 in ./venv/lib/python3.9/site-packages (from -r requirements.txt (line 5)) (4.3.4)
Requirement already satisfied: wrapt==1.14.1 in ./venv/lib/python3.9/site-packages (from -r requirements.txt (line 6)) (1.14.1)

Add materials for geo aggregation

Add materials for geo aggregations including the new polygon functionality when that is available. Also point to any external developer relations information that we produce as extra learning material.

Check exam question 1.4, Discord user reported that incorrect options are correct

From Discord:

Exam question 1.4 shows some of incorrect options as correct in the answer.

Book 204 gets deleted and is then needed.

Feedback from Discord:

in 1.4 we delete ru204:book:425, but in 3.1 we need to query the book (FT.SEARCH index:bookdemo "@title:aftertime") but of course doesn't exist anymore.

Look into this and correct as needed!

Update end of section 4 text in this repository.

Update the end of section 4 text in this repository to represent the final state, as the current version is the interim one.

Add XML files for final exam questions to this repository.

When they're ready, add the XML files for the final exam questions to this repository.

Update Jedis example to use latest release

The Jedis example is currently on a snapshot version of Jedis, update it to use the latest release when out (needed a command that wasn't released yet).

Move embedded images to S3

There are a few embedded image files in the HTML segments in this course. Move these to the S3 bucket for performance reasons, and update the links in the HTML docs.

Add materials for `JSON.MSET` command.

When the JSON.MSET command is available in the cloud, enhance the course with content for it.

Add materials for `JSON.MERGE` command.

When the JSON.MERGE command is available in the cloud, enhance the course with content for it.

Identical connection files used

util/connection.py and week4/util/connection.py both exist and have an empty diff. As such I can't imagine any good reason for having both of them. While I applaud the decision to localize the connection information in a single location, as opposed to other classes, it would nevertheless appear that some housekeeping is in order.

Add coupon code to Redis Cloud section

This is a reminder for us to add a coupon code RU204 to the Redis Cloud section of the installation Readme.

Section 3.10 - check index name

FT.CREATE index:bookdemo - this declares that we will be creating an index named ru204:book:index.
Please check the index name, it should read index:bookdemo rather than ru204:book:index

Add transcript for explainer video when we have it

When we have the transcript for the JSON explainer video that was newly added to section 1.2, add it in the learning management system and commit it to this repository.

Fix this in:

• 2023_01 run
• 2023_02 run
• SP run
• Back up changes to drive

Unable to load sample data using Docker

👋 Hello.

I'm having trouble loading the data using Docker. I've followed the instructions up until https://github.com/redislabs-training/ru204#load-the-sample-data-into-redis-and-create-the-search-index and, with the redisu-ru204 Docker container running, running python data_loader.py --dir data/books results in this stack trace:

$ python data_loader.py --dir data/books
Connecting to Redis at redis://localhost:6379
Deleting any existing JSON documents for RU204.
Dropping any existing search index.
Creating search index.
Traceback (most recent call last):
  File "/Users/cmoel/Desktop/ru204/data_loader.py", line 41, in <module>
    r.ft(INDEX_NAME).create_index(
  File "/Users/cmoel/Desktop/ru204/venv/lib/python3.11/site-packages/redis/commands/search/commands.py", line 139, in create_index
    return self.execute_command(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cmoel/Desktop/ru204/venv/lib/python3.11/site-packages/redis/client.py", line 1238, in execute_command
    return conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cmoel/Desktop/ru204/venv/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
           ^^^^
  File "/Users/cmoel/Desktop/ru204/venv/lib/python3.11/site-packages/redis/client.py", line 1239, in <lambda>
    lambda: self._send_command_parse_response(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cmoel/Desktop/ru204/venv/lib/python3.11/site-packages/redis/client.py", line 1215, in _send_command_parse_response
    return self.parse_response(conn, command_name, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cmoel/Desktop/ru204/venv/lib/python3.11/site-packages/redis/client.py", line 1254, in parse_response
    response = connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cmoel/Desktop/ru204/venv/lib/python3.11/site-packages/redis/connection.py", line 839, in read_response
    raise response
redis.exceptions.ResponseError: unknown command 'FT.CREATE', with args beginning with: 'idx:books' 'ON' 'JSON' 'PREFIX' '1' 'ru204:book:' 'SCORE' '1.0' 'SCHEMA' '$.author' 'AS' 'author' 'TEXT' 'WEIGHT' '1.0' 'SORTABL'

The error is caused by data_loader.py:41.

I've inspected the r variable in data_loader.py and it appears to be connected to the correct database in the container. Is there something in particular I should try next?

Update Justin's social link in the course intro

@justincastilla your Twitter link here isn't valid anymore, want to update this file with an alternative?

ru204/courseware/html/section_0/ru204_0_2_0_introduction_to_ru204.html

Line 3 in 508c42a

<p>We're <a href="https://simonprickett.dev/" target="_blank" class="page-link">Simon Prickett</a> and <a href="https://twitter.com/JustCastilla" target="_blank" class="page-link">Justin Castilla</a>, Developer Advocates at Redis, and we'll be helping you along the way as you progress through this course.</p>

Add HTML files for each text segment of the course to this repo.

Once they are all finished, add the HTML files for each text segment of the course into this repo so that they can be managed here.

Add more example for redis om .net

I want to add more examples for redis om .net .
Does it possible to contribute to this repo?

Update Redis OM Node materials to use embedded objects when this is added to Redis OM Node

Update the Redis OM Node materials and source code to use embedded objects when this has been added to Redis OM Node.

Tracking the change to Redis OM Node here: redis/redis-om-node#75

Update section 4.3.0 to reflect configurable Redis URL for .NET code sample.

Update the code sample here:

https://github.com/redislabs-training/ru204/blob/main/courseware/html/section_4/ru204_4_3_0_querying_json_documents_with_redis_om_for_dot_net.html

to reflect changes that make the Redis URL configurable (see #17).