owasp / opencre

Common Requirements Enumeration Application

See the application working and more explanation at https://www.opencre.org CRE is an interactive content linking platform for uniting security standards and guidelines. It offers easy and robust access to relevant information when designing, developing, testing and procuring secure software. This python web and cli application handles adding and presenting CREs.

WHY?

Independent software security professionals got together to find a solution for the complexity and fragmentation in today’s landscape of security standards and guidelines. These people are Spyros Gasteratos, Rob van der Veer and friends, in close collaboration with the SKF, OpenSSF and Owasp Top 10 project.

HOW?

The CRE links each section of a standard to a shared topic (a Common Requirement), causing that section to also link with all other resources that map to the same topic. This 1) enables users to find all combined information from relevant sources, 2) it facilitates a shared and better understanding of cyber security, and 3) it allows standard makers to have links that keep working and offer all the information that readers need, so they don’t have to cover it all themselves. The CRE maintains itself: topic links in the standard text are scanned automatically. Furthermore, topics are linked with related other topics, creating a semantic web for security.

Example: the session time-out topic will take the user to relevant criteria in several standards, and to testing guides, development tips, more technical detail, threat descriptions, articles etc. From there, the user can navigate to resources about session management in general.

Some of the data has been kindly contributed by the SKF and ASVS projects

Running

Locally

Docker

The easiest way to run OpenCRE locally is by running the published docker container. You can do so by running: docker run -p 5000:5000 ghcr.io/owasp/opencre/opencre:latest After the container has finished downloading the remote information you can access it in localhost

Command Line

To run outside of Docker you need to install OpenCRE. To install this application you need python3, yarn and virtualenv.

• Clone the repository:

git clone https://github.com/OWASP/common-requirement-enumeration 

• Install dependencies

 make install 

• Download the latest CRE graph from upstream by running

python cre.py --upstream_sync

Keep in mind that until Issue #534 is fixed you won't have access to gap analysis results locally

To run the CLI application, you can run

python cre.py --help

To download a remote cre spreadsheet locally you can run

python cre.py --review --from_spreadsheet < google sheets url>

To add a remote spreadsheet to your local database you can run

python cre.py --add --from_spreadsheet < google sheets url>

To run the web application for development you can run

$ make start-containers
$ make start-worker 

# in a seperate shell
$ make dev-flask

Alternatively, you can use the dockerfile with

make docker && make docker-run

Some features like Gap Analysis require a neo4j DB running, you can start this with

make docker-neo4j

enviroment varaibles for app to connect to neo4jDB (default):

• NEO4J_URL (neo4j//neo4j:password@localhost:7687)

To run the web application for production you need gunicorn and you can run from within the cre_sync dir

make prod-run

Docker

You can build the production or the development docker images with make docker-prod and make docker-dev respectively The environment variables used by OpenCRE are:

        - name: NEO4J_URL
        - name: NO_GEN_EMBEDDINGS
        - name: FLASK_CONFIG
        - name: DEV_DATABASE_URL
        - name: INSECURE_REQUESTS # development or TLS terminated environments only
        - name: REDIS_HOST
        - name: REDIS_PORT
        - name: REDIS_NO_SSL
        - name: REDIS_URL # in case REDIS_HOST and REDIS_PORT are unavailable
        - name: GCP_NATIVE # if there are ambient GCP credentials, only useful for VERTEX chatbot
        - name: GOOGLE_SECRET_JSON # if not running on GCP
        - name: GOOGLE_CLIENT_ID # useful for login only
        - name: GOOGLE_CLIENT_SECRET # useful for login only
        - name: LOGIN_ALLOWED_DOMAINS # useful for login only
        - name: ENABLE_TRACING
        - name: OpenCRE_gspread_Auth # useful only when importing data, possible values 'oauth' or 'service_account'

You can run the containers with make docker-prod-run and make-docker-dev-run

Developing

You can run backend tests with

make test

You can run get a coverage report with

make cover

Try to keep the coverage above 70%

Contributing

Please see Contributing for contributing instructions

Roadmap

For a roadmap of what we would like to be done please see the issues.

opencre's Issues

Mees, Make app responsive (looks unusable on iphone)

Hierarchy jumps are not being imported

558-807 for example has a hierachy 1, 2 and not a 3. It seems that is the reason that the link between that CRE and it's 'parents' are not being imported.
https://www.opencre.org/cre/558-807 has no 'is part of' and the particular cre is also not mentioned in the contains list of its parent https://www.opencre.org/cre/270-568

Data: complete standard hyperlinks and names

After having decided between v2 and v3:
add urls for CWE, OPC, NIST(Torsten) and WSTG, and nice names for the cheat sheets

Make frontend show new linktypes

Ensure reliability, performance ,security

Please ensure that opencre frontend, backend are reliable (high availability, resistent against dos issues), secure (availability and integrity) and performant (under high load when gaining popularity or at peeks (press launch), and dos).
When in doubt, can we make sure that we have a plan to upgrade or transition to another program or provider in case issues arise.

register new linktypes in the frontend

update const.tsx to recognize new linktypes

Create production docker container

start with distroless
https://medium.com/@luke_perry_dev/dockerizing-with-distroless-f3b84ae10f3a

Being able to search for text in names/descriptions and make it the default search type, ideally moving to one search type

Spyros, let standard items link to their own page

When clicking on a standard item, don't open the entire standard with all its' things. If you click on CRE-306, you want to be taken to a page that lists all linked items.
If people want to go to a standard, they can use the search bar for now.

debug frontend issues

Frontend has a couple bugs showing things, debug

make opencre.org https only

redirect domain to https, make flask redirect to https automagically

enable Flask compression

instructions https://pypi.org/project/Flask-Compress/

should improve page loading speeds back to green

Add search by tag in frontend

frontend needs to be able to search by tag

make frontend show Graph

there's an experimental "show CREs in a graph" frontend page, hook it up to the relevant rest calls

text search does not refresh search results after second search .

see title

Pagination doesn't work for NIST

Issue

What is the issue?

NIST v5 doesn't have pagination, but the request to the backend contains the page number

Expected Behaviour

when clicking on different pages they should show different results

Actual Behaviour

clicking on different pages, results stay the same

Steps to reproduce

visit https://opencreorg.herokuapp.com/standard/NIST%20800-53%20v5 and click on any page number

Remove the 'contains' sections from the nested views, and make the topics clickable, so user can go to the cre topic and see the contains there

Integrate frontend building to instructions and dockerfile

Create V2 Demo page

I don't see any NIST links, yet it's in the V2 sheet in column Q. We need it for evaluation

Set link types for data

Any standard entry is by default “linked”
Hierarchy above: “is part of”
Hierarchy across (other CRE topic or Tag): “related”
Hierarchy below: “Contains”

[frontend] make standards show hyperlinks to the original standard

standards do not show the hyperlink linking the standard entry with the third-party standard, fix this

Get rid of default LinkType:Same

with then new linktypes introduced in issue: 29 there is no need for Same to be the default anymore, remove it

Enable Postgres integration for heroku

heroku does not support sqlite,
Gotta migrate to their official postgres

Make text search the default search type and change its name from "CRE name" to "Topic text"

Nested views don't seem to be working for "related" items

Time management ticket: Distribute work between Spyros and Mees so that we can move to P1, P2 and P3. Rob and team focus on updates of source data

Mees, please build frontend on search function in backend (already merged)

https://opencreorg.herokuapp.com/rest/v1/text_search?text=session
It returns things with that text(A) and the things linked to it(B). The idea is to first show only the things with the text(A) and allow the user to click on them, with the regular click behaviour. This ignores the B items. Later, they can be used in a function that expands them for example. Layout of the result page can be the same as a CRE page.
Add an option "Text" to the dropdown of the search bar and make that the default one.

make heroku autodeploy from a master protected branch

Harold needs to add me as an OWASP org member

Make tags clickable and redirect to a page that searches by tag

Issue

What is the issue?

< answer here >

Expected Behaviour

What should have happened?

Actual Behaviour

What actually happened?

Steps to reproduce

How can we reproduce the error?

Urgent, for Mees: Improve the landing page content and look

The current landing page text needs to be improved in content and looks.
See for content, including hyperlinks: https://docs.google.com/document/d/1BqFwHYsA0R6DA_9HgdeBaoDGEp7a1jZhUHmH_MN5a_E/edit#heading=h.ozvtefgoah0x

Setup collaboration/feedback options for the community

Spyros I know that you have specific good ideas about this. This is my stab at what I think is required: At launch it's important for the community to share input and provide suggestions immediately, particularly for the data. It requires that the source data is provided in such a way that people can find the data that they have suggestions for and suggest changes.
Processing changes directly to production is not high priority I guess, but registering input and comments is important.

Add pre-commit hooks

add pre commit hooks for
black .
mypy --strict --exclude venv/ .
FLASK_APP=cre.py FLASK_CONFIG=test flask test | grep -i FAIL # there has to be a better way to ensure tests pass
if [[ $(FLASK_APP=cre.py FLASK_CONFIG=test flask test --coverage | grep TOTAL | awk '{print $6}' | tr -d "%" ) -lt 70 ]] then exit 1 fi # there has to be a better way to ensure coverage

Setup email forwarding in the domain to the whole team and change contact info on the landing page to [email protected]

Mees or Spyros: Ability to show all top level CREs (CREs without ispartof links) as great starting point to explore and drill down

Urgent: make sure opencre.org is indexed by Google

Currently our opencre.org cannot be found if you search "common requirement enumeration" in google. We need to find out how to make it work. Maybe it's because of the way we forward to the heroku app?

Put proposed short webtext in the readme.md of the repo as well, perhaps with some alterations and additions - I leave that up to you

enable flask caching for index

instructions https://flask-caching.readthedocs.io/en/latest/

Docker Improvements

build frontend
install deps for python without building them from scratch

offer production option where application runs in a distroless container
improve docker entrypoint to optionally run migrations on launch
document required env vars to point to database outside container

rob mentioned a bug introducing cycles, investigate

Mees, create two ways to link from an item.

An item can link to its cre page or to the hyperlink.
Clicking on the item name should go the cre page. After the item name, add a underlined text "(content") that links in a new tab to the hyperlink.
Example: ASVS -V2.2.5 (content).
Yes, you read it correctly: underlined. It's the only way I can think of that suggests it is a separate thing to click on. I'm open to suggestions.
Also when you click on an item to go to its CRE page, on that CRE page also show that content thing after the name.

Opencre no https (I understood we have something in place but do not see the results)

Mees, please create the nested and grouped view

See the CREdesign powerpoint for more details (specs in this issue prevail). Two things:
.
1)Group linked things(standards or topics) by link type, in this order (perhaps having the linktypes a lighter color, so the topic names stand out more):
a) linktype "Linked" (fyi these will be standards),
b) then "is part of" (fyi these are topics linked one level higher in the hierarchy, if any)
c) then "contains" (fyi these are topics linked one level lower in the hierarchy, if any)
d) then "related" (fyi these are topics cross-linked, if any)
.
2)Nest under "is part of" and "related" topics: the same overview of things with link types for that topic, and so on(recursively), excluding the "contains", indented.
.
Example 1:
CRE 482-866 "Encrypt personal data at rest"
..Linked: ASVS - V6.1.1
..Is part of: Securely store regulated data

....Is part of: Encrypt data at rest
......Linked: CWE - 311: (Missing Encryption of Sensitive Data)
......Is part of: Secure data storage
........Linked: OWASP Top 10 - A3 (Sensitive data exposure)
......Related: Cryptography

..Related: Personal data protection
....Linked: NIST SP800 53 - SC11
.
Example 2:
CRE 538-304 "Encrypt data at rest”

..Linked: CWE - 311: (Missing Encryption of Sensitive Data)

..Is part of: Secure data storage
....Linked: OWASP Top 10 - A3 (Sensitive data exposure)�

..Related: Cryptography

..Contains: Encryption algorithms
..Contains: Securely store regulated data
..Contains: Secrets shared with clients only for low risk

Make every page have a container instead of <body>

Issue

What is the issue?

search page and the various standards pages do not hace a container div, this means that when you scroll too much you see the body which is white and breaks colouring

Expected Behaviour

What should have happened?

Actual Behaviour

What actually happened?

Steps to reproduce

How can we reproduce the error?

Treat tags just the same as "link to other cre"

I noticed that many topics have related topics that are wrong, like for example "Encrypt data at rest" is related to "Communication encryption". After some analysis it seems that these relations have been created through tags. This is not how tags are intended. Tags are cre topics. So if the tag column mentions "Cryptography" it means that it links to the Cryptography topic and not to everyhting else that is tagged with Cryptography. Tags can be dealt with really simply: you can treat them just the same as you do the elements in the 'Link to other CRE' column. In fact the columns could be combined with eachother.

So instead of encrypt data at rest becoming related to MFA/OTP, secret storage etc. etc (all things that are tagged with cryptography), it only should be related to "Cryptography" and then when you click on cryptography, then you see all the other things.

make some more linktypes and make them implicit

Design document mentions a number of implicit Links (child of, parent of etc) make the relevant changes so that the correct linktypes get assigned when docs are exported from the DB

ensure when Docs are added to the DB lineage is preserved

Tags don't seem to work

In the tag column for https://www.opencre.org/cre/268-088 is DOS, but it doesn't show.
Also I don't see tags for XSS and Injection.
There is a topic for injection and it links to a top 10 entry, but injection does not link to any CREs.
Has the tag column not been imported? It basically is the same as the "other cre" column.
Also note that some tag cells have two entries, separated by commas

Spyros, Owasp top 10 entries have a hyperlink in the source, but do not take user to external page

http://www.opencre.org/standard/TOP10%202017

make all search case insensitive

Rename the tab name "document view" to "Open CRE" and when clicked let it go to the landing page

Issue

What is the issue?

< answer here >

Expected Behaviour

What should have happened?

Actual Behaviour

What actually happened?

Steps to reproduce

How can we reproduce the error?