scratchfoundation / scratch-parser Goto Github PK

View Code? Open in Web Editor NEW

59.0 31.0 59.0 20.43 MB

Validation and parsing for Scratch projects

License: BSD 3-Clause "New" or "Revised" License

JavaScript 100.00%

scratch-parser's Introduction

scratch-parser

Parser for Scratch projects

Overview

The Scratch Parser is a Node.js module that parses and validates Scratch projects.

API

Installation

npm install scratch-parser

Basic Use

var fs = require('fs');
var parser = require('scratch-parser');

var buffer = fs.readFileSync('/path/to/project.sb2');
parser(buffer, false, function (err, project) {
    if (err) // handle the error
    // do something interesting
});

"Info"

In addition to the _meta data described above, Scratch projects include an attribute called info that may include the following:

Key	Description
`flashVersion`	Installed version of Adobe Flash
`swfVersion`	Version of the Scratch editor used to create the project
`userAgent`	User agent used to create the project
`savedExtensions`	Array of Scratch Extensions used in the project

Testing

Running the Test Suite

npm test

Code Coverage Report

make coverage

Performance Benchmarks / Stress Testing

make benchmark

Committing

This project uses semantic release to ensure version bumps follow semver so that projects using the config don't break unexpectedly.

In order to automatically determine the type of version bump necessary, semantic release expects commit messages to be formatted following conventional-changelog.

<type>(<scope>): <subject>
<BLANK LINE>
<body>
<BLANK LINE>
<footer>

subject and body are your familiar commit subject and body. footer is where you would include BREAKING CHANGE and ISSUES FIXED sections if applicable.

type is one of:

fix: A bug fix Causes a patch release (0.0.x)
feat: A new feature Causes a minor release (0.x.0)
docs: Documentation only changes
style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
refactor: A code change that neither fixes a bug nor adds a feature
perf: A code change that improves performance May or may not cause a minor release. It's not clear.
test: Adding missing tests or correcting existing tests
ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
chore: Other changes that don't modify src or test files
revert: Reverts a previous commit

Use the commitizen CLI to make commits formatted in this way:

npm install -g commitizen
npm install

Now you're ready to make commits using git cz.

Breaking changes

If you're committing a change that will require changes to existing code, ensure your commit specifies a breaking change. In your commit body, prefix the changes with "BREAKING CHANGE: " This will cause a major version bump so downstream projects must choose to upgrade the config and will not break the build unexpectedly.

scratch-parser's People

Contributors

Stargazers

Watchers

scratch-parser's Issues

Stripping `\b` makes some projects not load.

Project 286851747 suffers from the backspace control character getting stripped out too aggressive. See scratchfoundation/scratch-gui#4767 (comment).

A possible solution is to not strip out \b if it is preceded by another \ (e.g. the user defined string in a list value in this project, 'have\\been')

Related issue

In this issue: scratchfoundation/scratch-www#3497 , @adroitwhiz gave the following description. @benjiwheeler was able to reproduce the problem.

If any string input/argument in any block in a project contains the substring "\b" anywhere, attempts to save the project to the Scratch server will result in an HTTP 500 error.

Other backslash sequences ("\n", "\t", etc.) seem to work fine--I've tested all letters of the alphabet (in lowercase) and only "\b" causes a 500 error.

Downloading the project as a .sb3 works fine, so this appears to be a website issue which is why I'm filing this bug against scratch-www (there's no public repo for the Scratch backend).

To Reproduce:

Type a string containing "\b" into any text input:

Observe the 500 error when saving:

Scratch 3 should allow sprites named "stage"

In both the 2.0 and 3.0 editors, you can create a sprite named 'stage'. Saving and loading this project in 3.0 does not work. This is the root cause of the not being able to remix chess-escape project.

E.g. saving and loading this project in 3.0 does not work:
https://scratch.mit.edu/projects/26069997

Project fails to parse for unknown reason

https://cdn.projects.scratch.mit.edu/internalapi/project/165661752/get/

/via @shrutiM15

Add tests for 3.0 format.

.json and .sb3 fixtures should get added for the 3.0 format and should be added to existing test suite.

Handle JSON with Infinity

https://scratch.mit.edu/projects/10590338/ is crashing because the project.json has Infinity (which is invalid).

SB3 monitors are missing?

I was using the SB3 schema and definitions to implement an SB3 serializer, but it seems there's no information on monitors there? This isn't particularly an issue for implementing the serializer, since I could find that info in scratch-vm, but it makes me wonder how complete of a reference scratch-parser is; irregardless, monitors seem like something that should be implemented here.

Handle errors when extracting ZIP archive

Cannot read property "bind" of undefined

Code:

const parse = require('scratch-parser')
const fs = require('fs')

let buff = fs.readFileSync('./project.sb3')
parse(buff, function(err,project) {
console.log(project)
})

What I get:

C:\Users\themysticsavages\desktop\getscratchscripts\node_modules\scratch-parser\index.js:25
        .then(callback.bind(null, null), callback);
                       ^

TypeError: Cannot read property 'bind' of undefined
    at module.exports (C:\Users\themysticsavages\desktop\getscratchscripts\node_modules\scratch-parser\index.js:25:24)
    at Object.<anonymous> (C:\Users\themysticsavages\desktop\getscratchscripts\index.js:5:1)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
    at internal/main/run_main_module.js:17:47

Does someone know why this happens?

Some 2.0 projects fail to validate

Maze Starter (#10128431) does not validate as a 2.0 project (and therefore doesn't load):

Relevant error:
"sb2Errors":[{"keyword":"required","dataPath":".info","schemaPath":"#/properties/info/required","params":{"missingProperty":"swfVersion"},"message":"should have required property 'swfVersion'"}]

Add code coverage badge to CI process and README

The test coverage for the parser is currently 100%. I'd like to keep it that way!

Validation error for sb2 because objName for stage is null

https://scratch.mit.edu/projects/414035099/

{"validationError":"Could not parse as a valid SB2 or SB3 project.","sb2Errors":[{"keyword":"type","dataPath":".objName","schemaPath":"#/properties/objName/type","params":{"type":"string"},"message":"should be string"}],"sb3Errors":[{"keyword":"required","dataPath":"","schemaPath":"#/required","params":{"missingProperty":"meta"},"message":"should have required property 'meta'"}]}

Fields should not be saved separately from inputs

The block definition has (among other irrelevant ones) two properties: inputs and fields. These correspond to the two different ways to enter data into a Scratch 3.0 block, i.e. via input or via field.

In the 3.0 editor, "inputs" mean the following (and maybe more):

Number slots.
String slots.
Boolean slots.
Dropdowns which accept blocks (round dropdowns).

"Fields" usually refer to dropdowns which do not accept blocks (square dropdowns).

In Scratch 2.0, the save format for a block was relatively simple and straightforward: [blockName, ...blockArguments]. blockArguments was simply a list of strings, numbers, and blocks. A block saved as ["mycoolblock", "apple"] simply meant that there was a value "apple" in the first slot, regardless of whether that slot was a dropdown or string slot.

Fundamentally, Scratch 2.0's save format makes it easy to change the behavior of an input without breaking existing projects. As an example, I'll refer to scratchfoundation/scratch-blocks#1418, which reads:

Proposal

Accept reporters to be dropped on the PROPERTY field of the "sensing_of" block:

In 2.0's format, this means toggling a flag somewhere which makes the dropdown accept blocks.

3.0's save format, however, makes this difficult. It would be required to change a field to an input; this does break existing Scratch 3.0 projects, because the save format distinguishes fields and inputs. (In theory and practice, all occurrences of the dropdown which was previously a field lose their data, so the now-input dropdowns are left empty -- they search for data in "inputs" but it is stored in "fields".)

My proposal to allow for changes such as the one described above is to do the following:

Merge inputs and fields together. The unified property should be an object with input/field names (ala PROPERTY, CONDITION) as keys and one of the following as values:

A string ("Hello, world!") or a number (42).
A reference to a reporter block, probably saved as something like {"block": "<block ID>"}.

Let scratch-vm assign values to fields or inputs, according to the block definition in scratch-blocks, and have the VM create any necessary shadow blocks. The scratch-parser module would not know whether PROPERTY means an input or a field; instead, the VM would look at the scratch-blocks definition of the block (sensing_of) and see whether there is a PROPERTY input or a PROPERTY field. Then, if necessary, the VM would create required shadow blocks (or anything similar).

Since the parser wouldn't know the difference between text inputs or dropdowns which accept reporters or those that don't, there wouldn't be any difficulty changing between those in block definitions in scratch-blocks, making implementing ideas such as the one suggested above easier and non-destructive.

Always return an error object

Right now errors returned through the callback interface from scratch-parser can be of three types: objects, strings, or Errors. As we have done in other repos it would be good to normalize this to always return an error object.

webpack build scratch-parser warning

when i add scratch-parser in scratch-gui, webpack builder reports warnings and the broswer reports error

the node is v7.2.0
npm is v3.10.9

my code:

const parser = require('scratch-parser');

add nodent-loader in webpack.config.js to parse nodent

module: {
    loaders: [
        {
            test: /\.js$/,
            loader: 'nodent-loader'
        }
    ]
}

the webpack warnings when import scratch-parser:

WARNING in ./~/.3.0.20@nodent/nodent.js
Module parse failed: 
scratch/node_modules/.3.0.20@nodent/nodent.js Unexpected character '#' (1:0)
You may need an appropriate loader to handle this file type.
| #!/usr/bin/env node
|
| 'use strict';
 @ ./~/.4.7.0@ajv/lib/async.js 116:26-48
 @ ./~/.4.7.0@ajv/lib/ajv.js
 @ ./~/.1.0.1@scratch-parser/lib/validate.js
 @ ./~/.1.0.1@scratch-parser/index.js
 @ ./src/scratch-extend/gui/src/containers/load-button.jsx
 @ ./src/scratch-extend/gui/src/components/menu-bar/menu-bar.jsx
 @ ./src/scratch-extend/gui/src/components/gui/gui.jsx
 @ ./src/scratch-extend/gui/src/containers/gui.jsx
 @ ./src/scratch-extend/gui/src/index.jsx
 @ ./src/index.js
 @ multi ./~/.2.4.2@webpack-dev-server/client?http://0.0.0.0:8601 ./src/index.js

WARNING in ./~/.6.25.0@babel-core/lib/transformation/file/options/option-manager.js
178:19-37 Critical dependency: the request of a dependency is an expression

WARNING in ./~/.6.25.0@babel-core/lib/transformation/file/options/option-manager.js
296:16-34 Critical dependency: the request of a dependency is an expression

WARNING in ./~/.6.25.0@babel-core/lib/transformation/file/index.js
500:24-39 Critical dependency: the request of a dependency is an expression

WARNING in ./~/.6.25.0@babel-core/lib/transformation/file/index.js
699:16-34 Critical dependency: the request of a dependency is an expression

the error in browser when start up the scratch-gui project:

Hope to help.

Scripts with '\bb' in a text input disappear

EDIT: this appears to only affect Firefox

This is another instance of ASCII backspaces making scripts disappear.

The sequence "\b" is stripped from the input JSON string before parsing. This is to prevent ASCII backspaces from appearing during parsing, which cause scripts to disappear.

However, when the raw text "\b" (a backslash followed by a lowercase b) appears inside double-quoted strings, it takes the form "\b" (a backslash followed by another backslash followed by a lowercase b). The second backslash escapes the first.

As such, when you type "\bb", it is serialized as "\bb". The parser strips out the "\b" in the middle, leaving one additional "\b". This is then parsed into an ASCII backspace, and the script disappears.

Serialize sb3 `blocks` section to a more compact format

In #21, we serialize the exact contents of blocks in vm, but for filesize concerns, we should make this section to be more similar to sb2's format.

Normalize / expand test suite

Our design patterns for test suites have changed quite a bit since developing this project. We should go back and clean up our tests and try to ensure that we get coverage back up to 100%.

Projects fail to upload if they've been rezipped with a folder

2.0 and 3.0 projects that have been downloaded, extracted, and rezipped as a folder fail to upload in Scratch 3.0. This is not true in Scratch 2.0, and should be fixed here and in the VM.

Add "complexity" metric to stats

For search, we'd like the ability to use project complexity as one attribute (among many) in our relevance (ranking) algorithm (particularly for new users). This requires a few things:

Prior Research

Review the overall relevance function as a group (see scratch-api)
Decide on what attributes could constitute "complexity" as a group. E.g.:
- Number of blocks
- Number of scripts
- Number of sprites
- Number of costumes
- Number of sounds
- Diversity of blocks
- Diversity of block categories
- Diversity of concepts (see: AREA 2012 paper)
- Presence of specific blocks / concepts (broadcast, procedures, etc.)
- Cyclomatic complexity
- etc.
Collect a random sample of projects
Define a loose rubric for evaluating the complexity of a project (see above)
Ask team to rank those projects in complexity from least to greatest

Proposal One

Develop a heuristic function that transforms these attributes into an aggregate "score"
Embed function into parser
Check against evaluation set

Proposal Two (Alternate)

Train a machine learning algorithm on a subset of those projects (training set)
Embed function into parser
Check against evaluation set

Questions

What about "artistic" complexity? Things like number of costumes, if the costumes were uploaded vs. drawn in Scratch, etc. all could have a big impact on the perception of complexity. (from @seotts)
In either approach how important is the "interpretability" of the complexity ranking function? For example, a NN approach may give us a tighter fit but would be difficult / impossible to describe. (from @jwzimmer)

References

Could not parse as a valid SB2 or SB3 project.

When I tried to load a .sb3 file, I encountered the following error.

lib.min.js:181

{
    "validationError":"Could not parse as a valid SB2 or SB3 project.",
    "sb2Errors":[
        {
            "keyword":"required",
            "dataPath":"",
            "schemaPath":"#/required",
            "params":{
                "missingProperty":"objName"
            },
            "message":"should have required property 'objName'"
        }
    ],
    "sb3Errors":[
        {
            "keyword":"enum",
            "dataPath":".targets[1].isStage",
            "schemaPath":"sb3_definitions.json#/definitions/sprite/properties/isStage/enum",
            "params":{
                "allowedValues":[
                    false
                ]
            },
            "message":"should be equal to one of the allowed values"
        }
    ]
}

And I modify the isStage's value to 'false', it can be worked...

So... how to solve it

Add integrations tests for 2.0 validation errors

Add integration tests for one of each kind of validation error caused by projects discussed in #27.

Can't convert to zip

I used a batch file to copy the file, but it won't convert to a zip file. Please allow zip conversions for block "hacking".
Batch file code:

@echo off
title "COPY A FILE"
color 1f
echo Welcome to the file copier!
PAUSE
:top
echo What file do you want to copy?
set /p fileone=
echo What will the new file name be?
set /p filetwo=
more %fileone%>%filetwo%
echo File Copied
PAUSE
echo "Would you like to copy another file? [Y/N]"
set /p choice=
if "%choice%"=="Y" (
goto top
)
if "%choice%"=="N" (
goto end
)
:end
echo Thank you for choosing File Copier to copy your files!
PAUSE
exit

I added the code in case it is a code error for the batch file.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Warning

These dependencies are deprecated:

Datasource	Name	Replacement PR?
npm	`babel-eslint`

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions

.github/workflows/deploy.yml

actions/checkout v4@b4ffde65f46336ab88eb53be808477a3936bae11

wagoid/commitlint-github-action v5@5ce82f5d814d4010519d15f0552aec4f17a1e1fe

actions/setup-node v3@1a4442cacd436585916779262731d5b162bc6ec7

npm

package.json

ajv ^6.3.0

jszip ^3.1.5

pify ^4.0.1

@commitlint/cli 7.2.1

@commitlint/config-conventional 7.1.2

@commitlint/travis-cli 7.2.1

babel-eslint 10.1.0

benchmark 2.1.4

cz-conventional-changelog 2.1.0

eslint 8.57.0

eslint-config-scratch 9.0.7

glob 7.2.3

husky 0.14.3

semantic-release ^15.1.6

tap 11.1.5

node >=8.0

nvm

.nvmrc

node v18

Check this box to trigger a request for Renovate to run again on this repository

[ { keyword: 'required',
    dataPath: '',
    schemaPath: '#/required',
    params: { missingProperty: 'sounds' },
    message: 'should have required property \'sounds\'' } ]

As many Scratch projects have sprites that don't have any sounds, and the "sounds" only gets exported if the target sprite has at least one sound, I don't think this should throw an error.

I could be wrong, because I haven't checked, but similar errors probably get thrown for other missing properties.

I've made a simple program to fix this issue: towerofnix/scratch-parser-stupid-issue-fixer

Incorrect Blocks count in _meta

Running the scratch-parser on a project resulted in an incorrect blocks count. The correct count should be 20 but the results shows 9.
The screenshots from the project and the results from running the scratch-parser are pasted below.

Support Scratch 1.4

Parser for .sb files
Converter for normalization between 1.4 and 2.0 projects
Test suite