adlnet / catapult Goto Github PK

Course:
Introduction to Geology - Multi AU

AU:
Quiz

The passed statement is sent before user attempts questionnaire.

I understand that this could be by design but it seems a little counter intuitive. I believe the user should at least attempt the quiz before the passed statement is sent.

Hi all,

Love the CTS, and it's great fun to use! While using it though I did come across the following oddity regarding setting mastery scores in the CTS test configuration screen (before you launch the course being tested). Please let me know if any additional details or information is needed!

Thank you.

Potential Issue Description 🔍

When modifying the AU's Mastery Score via the CTS it's possible to set the value beyond the expected 0~1 threshold.

For example, accidentally setting a Mastery Score to 90 (instead of a valid 0.9) will result in a test package's passing percentage to effectively be 9,000% instead of 90%, with no errors or issues exposed by the CTS.

Replication Steps 🔢

1. Either create, or access an existing course within the CTS with content that accepts and handles LMS modified mastery scores.
2. Either create, or access an existing test for the course.
3. In the Assignable Units section click on the Show Configuration toggle.
4. Enter a value greater than 1 for the Mastery Score
5. Launch the course content and take the quiz as expected. If authoring content does not have a handler for dealing with out-of-bound values provided by the LMS then it can result in bad passing requirements being passed to the test course.

Example screenshot of setting mastery score configuration value to 90.

Test cmi5 Package 📦

Rise 360 cmi5 test course package

Documentation 📚

cmi5 Mastery Score specification
CTS User Guide - Configuration settings

Hi All,

We have been struggling getting the CTS up and running, the player and rdbms services are running fine but not able to start webserivce

[dumb-init] ./entrypoint.sh: No such file or directory

Any help would help my team to upload and test the sample cmi5 packages

docker fails due to missing binaries at the URL https://github.com/kelektiv/node.bcrypt.js/releases/download/v5.0.1/bcrypt_lib-v5.0.1-napi-v3-linux-arm64-glibc.tar.gz.

Is there a work around for building cts from docker on an M1 Mac machine?

When sending a batch of statements (as in an array of more than one statement) with Initialized being the first of them, a false violation is produced: 9.3.0.0-4 - The "Initialized" verb MUST be the first statement (cmi5 allowed or defined).

The issue is in player/service/plugins/routes/lrs.js, which inspects each element of the statements array for: (! session.is_initialized && st.verb.id !== VERB_INITIALIZED_ID). Since is_initialized is only set later on, this code ends up outputting a false violation once it hits the second element of the array and beyond.

In 004-1-moveOn-Completed.zip, the 7th item indicates a requirement for a block Id to be included in the grouping for a Satisified statement. I can not find this requirement in the cmi5 spec.

In running AU 005-1-invalid-au, we receive an error citing requirement 9.1.0.0-1 (d). In reviewing the lrs, we accepted two statements, an Initialized (cmi5 defined) and an Experienced (cmi5 allowed). Both were sent by the AU with a statement id. I'm sure I am missing something, but it is hard to find without knowing what requirement fails.

Initialized:
{"id":"0cacc365-ed8d-4735-8000-80c8e9273529","timestamp":"2022-04-19T16:06:05.352Z","actor":{"objectType":"Agent","name":"Boss, Lisa J.","account":{"homePage":"https://www2.vta.com/Demo/LearnerConnection/","name":"27701-4"}},"verb":{"id":"http://adlnet.gov/expapi/verbs/initialized","display":{"en":"initialized"}},"object":{"id":"https://www2.vta.com/Demo/LearnerConnection/AU/13168"},"context":{"contextActivities":{"grouping":[{"objectType":"Activity","id":"https://w3id.org/xapi/cmi5/catapult/lts/au/005-1-invalid-au"}],"other":[{"id":"http://id.tincanapi.com/activity/software/@rusticisoftware/cmi5/3.0.0","definition":{"name":{"und":"@rusticisoftware/cmi5 (3.0.0)"},"description":{"en":"A JavaScript library implementing the cmi5 specification for AUs during runtime."},"type":"http://id.tincanapi.com/activitytype/source"}}],"category":[{"id":"https://w3id.org/xapi/cmi5/context/categories/cmi5"}]},"extensions":{"https://w3id.org/xapi/cmi5/context/extensions/sessionid":"2120"},"registration":"a98aef38-eda7-4985-ad97-780729746029"}}

Experienced:
{"id":"644558eb-67a8-4f33-8d7f-667cfedde421","timestamp":"2022-04-19T16:06:07.360Z","actor":{"objectType":"Agent","name":"Boss, Lisa J.","account":{"homePage":"https://www2.vta.com/Demo/LearnerConnection/","name":"27701-4"}},"verb":{"id":"http://adlnet.gov/expapi/verbs/experienced"},"object":{"id":"https://www2.vta.com/Demo/LearnerConnection/AU/13168"},"context":{"contextActivities":{"grouping":[{"objectType":"Activity","id":"https://w3id.org/xapi/cmi5/catapult/lts/au/005-1-invalid-au"}],"other":[{"id":"http://id.tincanapi.com/activity/software/@rusticisoftware/cmi5/3.0.0","definition":{"name":{"und":"@rusticisoftware/cmi5 (3.0.0)"},"description":{"en":"A JavaScript library implementing the cmi5 specification for AUs during runtime."},"type":"http://id.tincanapi.com/activitytype/source"}}]},"extensions":{"https://w3id.org/xapi/cmi5/context/extensions/sessionid":"2120"},"registration":"a98aef38-eda7-4985-ad97-780729746029"}}

Hi, we have some issues with the CTS, maybe you can help us with some.

1. Permission issues after initial setup

We start from a completely clean system:

cd cts
sudo rm -rf var
docker stop $(docker ps -aq)
docker rm $(docker ps -aq)
docker system prune  --all --force --volumes

Build the CTS as explained in the README:

# create .env file with the example content from the readme, then
docker-compose up --build -d

Then:

• open http://localhost:63399/
• create a user
• login with the user
• Go to New Course and upload a file

=> This fails:

Failed to import course: Error: Request failed: Failed request to player import: An internal server error occurred (Failed to create course content directory (/usr/src/app/plugins/routes/v1/../../../var/content/1/3): Error: EACCES: permission denied, mkdir '/usr/src/app/plugins/routes/v1/../../../var/content/1/3') (400)

We currently manually fix it like this:

docker-compose down
docker-compose run --rm -u root player chown -R node:node var
docker-compose up -d

Guess this should be fixed in the repo by adjusting the volumes.

2. LRS_ENDPOINT .env example values

Using the provided LRS_ENDPOINT="http://host.docker.internal:8081/20.1.x/lrs/default/" does not work on linux as this is windows only:

Failed to load learner preferences: Error: Request failed: An internal server error occurred (500 - Failed to request learner preferences from player (500): An internal server error occurred (Failed request to retrieve learner preferences: Error: Client request error: getaddrinfo ENOTFOUND host.docker.internal))

But as you are using docker-compose anyway we should be able to simply use docker-compose networking like this: LRS_ENDPOINT="http://player:3398" but this also fails when we try to test any content:

Failed to create session: Error: Request failed: An internal server error occurred (500 - Failed to retrieve AU launch URL from player (500): An internal server error occurred (Failed to store LMS.LaunchData state document (404): [object Object]))

Changing this to LRS_ENDPOINT="http://player:3398/lrs/default/" (as i guess this is the xapi base uri?) also does not work:

Failed to load learner preferences: Error: Request failed: An internal server error occurred (500 - Failed to request learner preferences from player (500): An internal server error occurred (Failed to retrieve learner preferences (401): [object Object]))

The error messages don't really help here. I guess right now the project is mostly lacking documentation (e.g. what are the routes of the player? where even is the lrs xapi base endpoint?) and logging. All projects do not have any request/error logging at all (besides the startup messages visible via docker logs) so debugging is really impossible right now.

Thanks!

Hi,

we use a fairly low default limit of 5 statements per query if you do not specify a limit. This leads to issues with some tests in the LTS, the first one being https://github.com/adlnet/CATAPULT/blob/main/lts/__tests__/runtime.js#L95 because it expects a result with at least 6 statements on one page.

If a test expects more then one statement we should explicitly specify a limit to prevent this.

Edit: i think you are theoretically allowed to limit every request to just 1 result, regardless of what limit is provided by the client. So the better solution might be to follow the more property when we expect more then one statement?

Python, make and g++ dependency are required.

There are two errors in the xml file.
a) The au IRI is invalid.
b) The url is invalid as it is a relative path that is not part of a zip package.

I don't think this is a good test of the LMS import. The LMS may reject the package for one reason or the other, so it is not a good test of the IRI validation.

Steps to reproduce:

1. Clone repository
2. Change to ./player folder
3. Add .env file
4. Execute docker-compose up -d --build
5. Open http://localhost:3398 and http://localhost:3398/v1/api in browser
6. Both return: {"statusCode":404,"error":"Not Found","message":"Not Found"}

Content of .env file:

HOST_PORT=3398
CONTENT_URL=http://localhost:63398/content
API_KEY="apikey"
API_SECRET="apisecret"
TOKEN_SECRET="asdasdasdasd"
LRS_ENDPOINT="http://lrs.example.com/data/xAPI"
LRS_USERNAME="username"
LRS_PASSWORD="password"

webservice logs:

webservice_1           | Waiting for rdbms:3306..
webservice_1           | Connected!
webservice_1           | Already up to date
webservice_1           | [nodemon] 2.0.15
webservice_1           | [nodemon] to restart at any time, enter `rs`
webservice_1           | [nodemon] watching path(s): index.js knexfile.js lib/**/* plugins/**/*
webservice_1           | [nodemon] watching extensions: js,mjs,json
webservice_1           | [nodemon] starting `node index.js`
webservice_1           | Waiting for rdbms:3306.
webservice_1           | Connected!
webservice_1           | Waiting for rdbms:3306.
webservice_1           | Connected!
webservice_1           | Catapult player service running on http://62a8ef710777:3398

Does anyone have the same problem?

In https://github.com/adlnet/CATAPULT/blob/main/lts/__tests__/runtime.js#L134 you check that the fifth statement (expect(regStatements[4].verb.id)) uses the satisfied verb.
In https://github.com/adlnet/CATAPULT/blob/main/lts/__tests__/runtime.js#L143 you expect the sixth statement (expect(regStatements[5].verb.id)) to also use the satisfied verb, but the sixth statement will be the terminated statement.

The spec clearly states that "The LMS SHOULD NOT issue multiple statements with "Satisfied"", so i guess this is an error in the test suite unless i am missing something?

We are currently trying to setup the LMS Testsuite. It would be nice if you could simply test the provided player with the lts to have a working starting point for implementing your own lms.custom.js to test your own lms.

Right now the provided example .env files in the respective README.md of player and lts are quite different and even use different names and terminology for what is (supposably) the same thing.

What i would like to see is some kind of "unification" of both examples so users can simply copy/paste both .env files, start the player and run the lts against it.

Is this possible or am i missing something here?

I did setup catapult with cloud lrs.io

more or less working but I have strange issue in with call to GET /api/v1/sessions/21/lrs/activities/state

Added few console.logs at cts/service/plugins/routes/v1/sessions.js

                            if (req.method === "get" && req.params.resource === "activities/state") {
                                if (req.query.stateId === "LMS.LaunchData") {
                                    console.log(uri)
                                    console.log(rawProxyResponsePayload.toString())

on package initialize I'm getting:

Catapult CTS service running on http://4bebdc12c5c3:3399
http://player:3398/lrs/activities/state?agent=%7B%22name%22%3A%22Johnnie%20Schinner%22%2C%22account%22%3A%7B%22name%22%3A%2269d917f%22%2C%22homePage%22%3A%22urn%3Acatapult-cts%3A8bff369%22%7D%2C%22objectType%22%3A%22Agent%22%7D&activityId=https%3A%2F%2Fw3id.org%2Fxapi%2Fcmi5%2Fcatapult%2Fplayer%2Fcourse%2F3d318298-8ad8-468e-a301-3bdb15c62131%2Fau%2F0&stateId=LMS.LaunchData&registration=f8cadac8-30cb-4064-b5b9-a3a0d142c297
! ��Φ��,x�xD�"�V!�P��mY�X��i ɷ_����7�\>j���l�!�£q��a�>�;�5�1Ta0;~��0rF�����L���<L�
                                                                                  uN/VM��auƝ�/d\C}�!��Wi�������uГ�>m�c��D#\���x�bFZ$q�R(1nm�����K={/f���r^�ugDE\�K/�e׳�^�.�k���@����P�k��
               �����io��o���$�����_q����o��q5
Debug: internal, implementation, error 
    SyntaxError: Unexpected token ! in JSON at position 0
    at JSON.parse (<anonymous>)
    at handler (/usr/src/app/plugins/routes/v1/sessions.js:505:64)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async exports.Manager.execute (/usr/src/app/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at async Object.internals.handler (/usr/src/app/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at async exports.execute (/usr/src/app/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at async Request._lifecycle (/usr/src/app/node_modules/@hapi/hapi/lib/request.js:372:32)
    at async Request._execute (/usr/src/app/node_modules/@hapi/hapi/lib/request.js:280:9)

But strange thing is - if I do copy as CURL and repeat it from postman - it works fine and show:

http://player:3398/lrs/activities/state?agent=%7B%22name%22%3A%22Johnnie%20Schinner%22%2C%22account%22%3A%7B%22name%22%3A%2269d917f%22%2C%22homePage%22%3A%22urn%3Acatapult-cts%3A8bff369%22%7D%2C%22objectType%22%3A%22Agent%22%7D&activityId=https%3A%2F%2Fw3id.org%2Fxapi%2Fcmi5%2Fcatapult%2Fplayer%2Fcourse%2F3d318298-8ad8-468e-a301-3bdb15c62131%2Fau%2F0&stateId=LMS.LaunchData&registration=f8cadac8-30cb-4064-b5b9-a3a0d142c297
{"launchMode":"Normal","masteryScore":1,"moveOn":"CompletedAndPassed","contextTemplate":{"contextActivities":{"grouping":[{"id":"https://github.com/xapijs/cmi5-demo/cmi5-demo"}]},"extensions":{"https://w3id.org/xapi/cmi5/context/extensions/sessionid":"973a1ea5-0dd2-473a-9933-566a811a3ddd"}},"entitlementKey":{},"returnURL":"https://localhost/api/v1/sessions/__sessionId__/return-url"}

When using certain LRS', such as YET XAPI LRS, if you don't remove '/' from LRS.js in the line pasted below, then the LRS will not work with CATAPULT CTS:

                    const uri = `${req.server.app.lrs.endpoint}**/**${req.params.resource}${req.url.search}`,

Removing the / and re-building makes it work. Can this be remedied? Thank you!

The manual test procedure has two validation checks for 006-LaunchMode.zip. One for Browse and one for Review. I can only locate the first AU. Have these been combined or am I overlooking it. The AU message reads that "Expected launchmode to be Browse or Review..." so I'm not sure the second AU is necessary but it is listed as a requirement for conformance.

Why does the cmi5.xml for the "essentials" package use CDATA for the au url? What is the associated spec requirement?

We have trouble understanding how to run 007-1-multi-session.zip and 007-2-multi-session.zip as explained in https://github.com/adlnet/CATAPULT/blob/main/lts/procedure.md. The explanation has two stages: 1) AU Execution - Launch AU 0 in the package 2) AU Execution - Launch AU 0 in the package again using the same registration.

Does that mean that the test packages must be run twice with the same registrationId? We tried that but the result turned out to be following:

• First run creates the registrationId and launches the test package. It gives the expected result(success is true, isError is false). The last statement the test package sends is however http://adlnet.gov/expapi/verbs/terminated.
• Second run uses the same registrationId, but when it sends the http://adlnet.gov/expapi/verbs/initialized statement then is is stopped by our validation since the https://github.com/AICC/CMI-5_Spec_Current/blob/quartz/cmi5_spec.md#verbs_terminated states that: "The LMS MUST use the "Terminated" statement to determine that the AU session has ended. Upon receiving a "Terminated" statement, the LMS MUST wait a specified period of time (defined by the LMS implementation) after which it MUST reject statements (of any kind) for the AU session."

Our understanding is that the "AU session" is the same as the requests made from a AU with the same registrationId. In such case it should not be possible to send any statements with the same registrationId after the http://adlnet.gov/expapi/verbs/terminated has been sent.

In https://github.com/adlnet/CATAPULT/blob/main/lts/pkg/src/005-1-invalid-au/005-1-invalid-au.js#L763 we send the same Statement as in https://github.com/adlnet/CATAPULT/blob/main/lts/pkg/src/005-1-invalid-au/005-1-invalid-au.js#L345 again.

Per the xAPI spec:

An LRS MUST NOT make any modifications to its state based on receiving a Statement with a statementId that it already has a Statement for. Whether it responds with 409 Conflict or 204 No Content, it MUST NOT modify the Statement or any other Object.

We currently return a 409 Conflict but the CATAPULT Testsuite fails here.

Hi, when i run the docker-compose in the CTS i noticed that the docker_nginx is always in RESTARTING status and the docker_certbot in the EXITED status. Is this expected?

This is the modifications I did after cloning the repo.

cts/.env

cts/docker-compose

I can still view the CATAPULT CTS client http://127.0.0.1:3399/client/#/ though and can upload a course (though i couldn't launch it see #40).

In execution of 005-1-invalid-au, a call is made to the agent profile API. The call does not include an agent and is rejected by the LRS, making it impossible to complete the test.

Call is made to:
lrs_endpoint//agents/profile?profileId=cmi5LearnerPreferences

Note: Not a typo, the call has two slashes.

Payload is:
{languagePreference: "en-US,fr-FR,fr-BE", audioPreference: "on"}

Our version was downloaded 12/9/21. Is there a newer version?

The test expects in one of the items of this test to pass when the Auth sent in the header is empty. The CMI5 engine will reject it with a 400 return code. However, the testsuite expects it to accept it.

Looking into the testsuite, it is setting the Auth to an empty string

if (e.setAuth(""), !await i.sendStatement(e, r, "8.1.2.0-5 (d)")) return;

This is ultimately more of a question than anything else, but could potentially be an issue.

When initially running LTS against our LMS, many of the tests failed due to a timeout while waiting for this action in getResults inside lts/lib/helpers.js: await page.waitForSelector("#result");

When looking at the content manually in a browser, however, I found that the element the tests were searching for was present, but it was inside an iframe. For whatever reason, puppeteer wasn't finding any elements within the iframe on the page.

I modified the getResults function to this implementation which waits for the iframe, then gets the frame and checks it for the results element. If that fails, it reverts back to the original implementation and checks the main page for the results element, and if that fails, it reports the exception.

The question is -- was that OK to do, or should the original implementation not have had any issues with finding the results, and if it does then that's a problem with our LMS? I always have to assume that getting tests to pass by changing details of the test is not typically the right thing to do.

Below is my implementation of the getResult function, complete with wonky try/catch inside a catch 😅

getResult: async () => {
        let resultElement;

        const frameElement = await page.waitForSelector("iframe"),
            frame = await frameElement.contentFrame();

        try {
            resultElement = await frame.waitForSelector("#result");
        }
        catch (ex) {
            try {
                resultElement = await page.waitForSelector("#result");
            }
            catch (ex2) {
                throw new Error(`Failed to get result from content:\n Outer Ex: ${ex};\n\t inner ex: ${ex2}`);
            }
        }

        const jsonString = await resultElement.evaluate(el => el.textContent);

        let result;

        try {
            result = JSON.parse(jsonString);
        }
        catch (ex) {
            throw new Error(`Failed to parse JSON from string: '${jsonString}'`);
        }

        return result;
    },

Part of the content specification outlines that The AU MUST retrieve the Learner Preferences document from the Agent Profile on startup..

The tester shouldn't need to setup the agent profile in the client/#/test/{ID} route. It should automatically at least put a "default" document when then agent is created (new test in the CTS).

Hi, we have some issues with 005-1-invalid-au.zip, maybe you can give us some hints.

1. send statement before loading learner prefs

1.1

The requests to retrieve the cmi5LearnerPreferences does not contain any session or AU information at all. I think there is no way for an LMS to check if the learner preferences were requested for this specific AU, at least not if users are potentially able to execute multiple AUs in parallel. How are we supposed to solve this?

1.2

Where does this requirement even come from? The spec only says that the AU MUST request the learner preferences. It does not say that the LMS actually has to verify this.

2. save learner prefs with invalid language preference values

We currently return a 400 BAD REQUEST for invalid learner preferences but the testsuite fails. It shows the following error message:

4.2.0.0-2 - The LMS MUST have an account which is able to retrieve all Resource data (from the Statement API, etc, including attachments and extensions) about another distinct user across multiple sessions for that user.: Save learner preferences response status code: 400  (Testing 11.0.0.0-5 (d))

This error message has nothing to do with the specific test so we are not sure what is wrong. Also, something that we noticed through the whole testsuite: it tells you that (in this case) a 400 is the wrong status code. But it never tells you what the correct one would be, why?

The package contains two errors:

1. Relative url in a non-zipped package.
2. Query string conflict.

This is not a good test as the LMS may detect one error or the other and reject the package, but the test does not ensure that the query string conflict causes the rejection.

I'm encountering this error when trying to launch a course I uploaded (from course_package) in the AU > Launch

Steps to reproduce:

1. Launch the CTS client http://127.0.0.1:3399/client/#/ and login
2. Click New Course and upload a course from the course_package directory
3. Test the uploaded course, create an Actor
4. In the Assignable Units Pane, click Launch

This is my cts/.env configuration

I've read in the other issues that it may need to install an actual LRS, is this also the case here?

They will not leave if clicked on or ignored. This is happening in student view.

I had Schweta clear her cache, and try resigning into Moodle. The issue persisted. IT does not seem to happen to her in Firefox.

While opening any cmi5 course in the CTS player I catch CORS exception.

If there can be issue of wrong .env setup here I`m providing my settings that I used for docker compose:

SQL_PORT=3306
HOST_PORT=3399

HOSTNAME=localhost

PLAYER_HOST_PORT=3398
PLAYER_CONTENT_URL=http://localhost:3398/content
PLAYER_KEY='apikey'
PLAYER_SECRET="APIsecret"
PLAYER_TOKEN_SECRET="asdasdasdasd"

LRS_ENDPOINT="https://sample2-lrs-bujocli.lrs.io/xapi/"
LRS_USERNAME="admin"
LRS_PASSWORD="admin"

UPD: there is was a mistale in my env file (I wrote HOSTNAME:localhost instead of HOSTNAME=localhost ). Now env file is correct but CORS problem is still present

What
The sample AU's should use the “progressed” verb for indicating progress during a session. The cmi5 website suggests that this is a best practice.

AICC/CMI-5_Spec_Current#765 may contain some relevant information.

Why
Many Learning Management Systems (LMS) indicate progress to their users. It would benefit vendors that are adding cmi5 support to their LMS's.

Currently the sample courses expect that the endpoint URL does not contain a trailing slash. If the endpoint URL does contain a trailing slash then the course will make requests with URLs that contain double slashes.

For example:

https://example.com/xapi//activities/state?stateId=LMS.LaunchData

(URL shorted for readability)

If the LRS uses a HTTP firewall it will reject requests that contain a double slash.

See issue 710 in the cmi5 repository for more details.