canonical / ubuntu-com-security-api Goto Github PK

View Code? Open in Web Editor NEW

15.0 16.0 8.0 1.12 MB

The API for CVEs and USNs data.

Shell 12.47% Dockerfile 0.67% Python 86.06% Mako 0.27% HTML 0.54%

web-and-design

ubuntu-com-security-api's Introduction

ubuntu.com security API

API functions under ubuntu.com for querying CVEs and security notices.

Local development

The simplest way to run the API locally is using the dotrun snap:

dotrun  # In the root of the project folder

This will start a database, import some sample data and run the server. Exiting the server with ctrl + c should automatically stop the database again.

Once the server has started, you can query CVEs, Notices or Releases from the API:

Or view the API documentation at http://0.0.0.0:8030/security/api/docs.

Managing the project

It's best to run and manage the project using dotrun if possible. This will install pip dependencies automatically, and will also include any expected system level dependencies.

Dotrun commands

A number of "scripts" are defined in package.json for running with dotrun. These could usually also be run with yarn run {scriptname}.

dotrun start: Run serve (this is the same as simply running dotrun)
dotrun serve: Start the database and the API service
dotrun start-db: Start and attach to the database without starting the API
dotrun delete-db: Stop and delete the database
dotrun test: Run lint-python, and then test-python
dotrun lint-python: Check the format of the Python code
dotrun format-python: Automatically format the Python code with black
dotrun test-python: Run Python tests to check the API functionality
dotrun clean: Delete databases, containers and temporary development files

API and database management scripts

There are also some extra Python scripts to help with manipulating the API and database. There can also be run through dotrun:

dotrun exec scripts/create-cves.py scripts/payloads/cves.json  # Create a new CVE through the API
dotrun exec scripts/create-notice.py scripts/payloads/usn-4414-2.json  # Create a Notice through the API
dotrun exec scripts/create-release.py scripts/payloads/testy.json  # Create a Release through the API
dotrun exec scripts/delete-cves.py CVE-2019-20504  # Delete a CVE
dotrun exec scripts/delete-notices.py USN-4414-2  # Delete a notice
dotrun exec scripts/delete-release.py testy  # Delete a release
dotrun exec scripts/generate-sample-security-data.py  # Fill the database with thousands of fake records

Flask scripts

There are additionally some flask scripts to run needed database modifications.

flask insert_numerical_cve_ids # For each cve in the database, update the numerical_id column. Can be run repeatedly.

ubuntu-com-security-api's People

Contributors

Stargazers

Watchers

Forkers

albertkol jkfran chanix95 mtruj013 isabella232 carkod pfsmorigo tuankiet97

ubuntu-com-security-api's Issues

Incorrect CVES addressed value on /16-04

Summary

The number of CVE's addressed on https://ubuntu.com/16-04 is 8,142 but it should be around 5941. This number is generated by a query to the API - could there be a bug with the API when it tries to filter the correct CVEs?

Screenshot

Improve documentation to list the enum values

Summary

Q4. Can we document the expected response values and/or examples for each response key?
The model documentation doesn't tell me the expected set of values for response keys which I expect are probably enums such as the CVE "status" values. Without an doc somewhere describing expected output values for known keys, we risk the client being unable to parse/match values.

List of enums:
pocket: security, updates, esm-infra, esm-apps (can be null)
component: main, universe (can be null)
release_codename: see the list attached in the cve file. codename column.
cve.status: not-in-ubuntu, active, rejected
cve.packages.statuses.status: released, DNE, needed, not-affected, deferred, needs-triage, ignored, pending
priority: unknown, negligible, low, medium, high, critical

Investigate API stability

CVE API query [ https://ubuntu.com/security/cves.json ] on package "linux" with limit 20 or without any limit results in "504 Gateway Time-out" , it only works with limit of 10 or less than 10

Summary

CVE API query [ https://ubuntu.com/security/cves.json ]
on package "linux" with limit 20 or without any limit results in "504 Gateway Time-out" , it only works with limit of 10 or less than 10

Process

curl -s --connect-timeout 600 -m 600 -X GET -H "Content-Type: application/json" "https://ubuntu.com/security/cves.json?package=linux&limit=20"
504 Gateway Time-out
The server didn't respond in time.

curl -s --connect-timeout 600 -m 600 -X GET -H "Content-Type: application/json" "https://ubuntu.com/security/cves.json?package=linux&limit=10" | python -m json.tool | grep "^ "id"
"id": "CVE-2021-0920",
"id": "CVE-2021-3736",
"id": "CVE-2021-43389",
"id": "CVE-2020-27820",
"id": "CVE-2021-34981",
"id": "CVE-2021-43267",
"id": "CVE-2021-3892",
"id": "CVE-2021-43057",
"id": "CVE-2021-43056",
"id": "CVE-2021-3760",

Current and expected result

Here I expected to get a list of 20 CVEs, the latest ones, but instead I got "504 Gateway Time-out"
curl -s --connect-timeout 600 -m 600 -X GET -H "Content-Type: application/json" "https://ubuntu.com/security/cves.json?package=linux&limit=20"
504 Gateway Time-out
The server didn't respond in time.

Unify import and api schemas

Summary

We have an import schema for CVE and Notices. The API uses also a schema to return CVE and Notices. Ideally they are the same schema for each CVE and Notices.

Github actions break on installing `dotrun`

Summary

Update the dotrun installing Github actions to use canonical/install-dotrun@main just like on ubuntu.com: https://github.com/canonical-web-and-design/ubuntu.com/blob/main/.github/workflows/pr.yaml#L55-L56

That should fix the checks.

Add LSN import endpoint to API

We already have the endpoint on ubuntu.com, we should have it on the API too.

Add ROS ESM packages to Ubuntu CVE Tracker website

Summary

Please add ROS ESM packages to Ubuntu CVE Tracker website.

Process

N/A

Current and expected result

ROS ESM packages were recently added to UCT (see https://git.launchpad.net/ubuntu-cve-tracker/tree/ros-esm-xenial-kinetic-supported.txt and https://git.launchpad.net/ubuntu-cve-tracker/tree/ros-esm-bionic-melodic-supported.txt).
Please list these packages in the public UCT website, to publicly reflect any CVEs affecting them.

Screenshot

N/A

Browser details

N/A

Securty API: pocket or component origin needed for esm-infra versus esm-apps sources in package

Summary

krb5 packages statuses on trusty report an empty pocket: null and component: null for a package that is delivered by "esm-infra".
https://ubuntu.com/security/cves/CVE-2020-28196.json

Current and expected result

Expected values for either pocket or component delivered by ESM would be "esm-infra" or "esm-apps" in the database.

The ubuntu-advantage-client will rely on some indicator in CVE.packages.statuses the tell it what Ubuntu Advantage service needs to be enabled in order to obtain updates for that package.

[Describe what happened and what you expected.

USN dashboard page lists related USNs for USN-4754-4. Can related USNs be made available security/notices.json and/or security/notices/{notice_id}.json

Summary

Looking to inform ua fix <USN> callers about related USNs in the event they try fixing an inapplicable USN for their environment but where a related USN did affect their system.

In the case where a customer tries to fix `usn-4754-4 on a trusty/hirsute/groovy system, USN..-4 only represents Xenial and Bionic fixes. But USN-4754-1 does represent T, X, B, F, G,H releases and we could better inform them to check related USNs or ua client could check this itself to better fix this USN on a system.

USN-4754-4 vs USN-4754-1 are examples of partial ubuntu support matrices
curl -X GET curl https://ubuntu.com/security/notices/USN-4754-4.json | jq .

Trusty, hirsute, groovy components are represented here

curl -X GET curl https://ubuntu.com/security/notices/USN-4754-1.json | jq .

Without this USN relationship response from the USN API routes, API consumers would need to make additional API requests to each CVE in order to obtain the related USNs from the CVE.

Process

[List the steps to replicate the issue.]

Current and expected result

Would like to see a something like a related_notices key in usn responses.

This related is already published somehow on USN landing pages under "Related notices" https://ubuntu.com/security/notices/USN-4754-1

Add support for all binaries affected by a CVE fix on the USN API

Summary

As explained in canonical/ubuntu-pro-client#1445 (comment), when preparing Ubuntu Security Notices the security team will often attempt to prune the list of binary packages included in the USN to the set that we believe were affected by the vulnerability and that subset is what is submitted to the web teams security/notices/ API. This is the information consumed for presentation purposes on https://ubuntu.com/security/notices.

The ua client also consumes this information and cannot resolve situations where such not-submitted binaries are installed on a given system but the USN does not provide information about them (i.e.: the ua client needs information to take the decision).

Process

On a trusty system with an affected libpython2.7 installed and ua client configured, run:
$ ua fix CVE-2021-3177

Current and expected result

Current result:

$ ua fix CVE-2021-3177
CVE-2021-3177: Python 2.7 vulnerability
https://ubuntu.com/security/CVE-2021-3177
2 affected packages are installed: python2.7, python3.4
Error: CVE-2021-3177 metadata defines no fixed version for libpython2.7-stdlib.
2 packages are still affected: python2.7, python3.4
✘ CVE-2021-3177 is not resolved.

Expected
$ ua fix CVE-2021-3177 should resolve the CVE

Add validation error when status is not passed in version query

Summary

API is not filtering CVEs based on version, nor does it display an error message

Process

Visit https://ubuntu.com/security/cves.json?version=eoan (also try with different versions) and see that total_results remains unchanged

Current and expected result

Filtering should work like it does on the live site (which is currently not using the API)

CVEs website with links with extra text gets a broken redirection

Summary

Hi.
When we have a CVE that has a link with extra text in parenthesis in either the Bugs or References sections, it gets an incorrect link redirection because it adds the extra text to the links.
We are wondering if that is something that might be fixed, please.

In summary, when we have a http://link.to.something (notes) we would like to have the link shown in the page as http://link.to.something (notes) but the redirection to http://link.to.something

Example: https://ubuntu.com/security/CVE-2014-4607

Process

Access a CVE with links in Bugs and/or References with extra text in parenthesis and try to access that link.

Current and expected result

From the example above (https://ubuntu.com/security/CVE-2014-4607)

Current web source

<li><a href="https://bugs.launchpad.net/ubuntu/+source/krfb/+bug/1352421 (krfb)">https://bugs.launchpad.net/ubuntu/+source/krfb/+bug/1352421 (krfb)</a></li>

Expected web source

<li><a href="https://bugs.launchpad.net/ubuntu/+source/krfb/+bug/1352421">https://bugs.launchpad.net/ubuntu/+source/krfb/+bug/1352421 (krfb)</a></li>

Extra comments

There was a python pseudocode suggestion that was used to discuss this issue with @mtruj013 in mattermost some days ago that I would like to add here in case it helps:

{% for reference in cve.references %}
    reference_link = reference
    
    {% if ' (' in reference  %}
        reference_link = reference[0:reference.index(' (')].strip()
    {% endif %}
    
    <li><a href="{{ reference_link }}">{{ reference }}</a></li> 
{% endfor %}

....

{% for bug in cve.bugs %}
    bug_link = bug

    {% if ' (' in bug  %}
        bug_link = bug[0:bug.index(' (')].strip()
    {% endif %}

    <li><a href="{{ bug_link }}">{{ bug }}</a></li>
{% endfor %}

Allow `notice` with `SSN` prefix.

Summary

Allow notice with SSN prefix.

Update the Marshmallow schema Regex validation on the notice id to also allow SSN.

Add new `pocket` type `soss`

Summary

Add new pocket type soss.

Investigate and Optimise API memory problem

Summary

The API on production has memory problems. FIX IT!

Use Schema object instead of Dictionary inside `use_kwargs` to specify expected parameters

Find a way to make the expected args of get_cves and get_notices schema objects instead of dictionaries.

It seems like the package webargs allows you to use a schema inside use_kwargs, but either flask-apispec or apispec fails converting the schema into parameters when generating the documentation.

Add NVD/CVSS Details

Update payload as requested in this issue

The API filter "status" isn't working

Summary

When I try to filter the CVEs by status, there is always an error message and the results aren't retrieved.

Process

All these API calls currently fail

https://ubuntu.com/security/cves.json?status=active

{"errors":"{'query': {'status': {0: ['Cannot find a status with status active']}}}","message":"Invalid payload"}

https://ubuntu.com/security/cves.json?status=rejected

{"errors":"{'query': {'status': {0: ['Cannot find a status with status rejected']}}}","message":"Invalid payload"}

https://ubuntu.com/security/cves.json?status=not-in-ubuntu

{"errors":"{'query': {'status': {0: ['Cannot find a status with status not-in-ubuntu']}}}","message":"Invalid payload"}

Current and expected result

The API fails to get any result.

It's expected that the filter works properly.

Browser details

Brave browser: [Versión 1.43.89 Chromium: 105.0.5195.102 (Build oficial) (64 bits)]

Update notice endpoint to include related notices in returned payload

API response not in plain text format

Summary

I'm working on an application that consumes the Canonical security API. The interaction between the application and the API was working flawlessly from various months (a year, maybe) ago until last week when the API started to respond with a binary payload.

The application is made in C++ and uses libcurl to perform the HTTP queries. And the API URL from where the content is downloaded is https://ubuntu.com/security/cves.json.

IMPORTANT: The binary payload response is not done always, but in a random way. Sometimes the payload has the expected format (JSON) and sometimes it is a binary. Anyway, if I try to download the whole security feed by performing various HTTP queries, it's very likely that one of them will receive the binary payload.

Apparently, the binary payload comes compressed in br format. In the image below, you can see the HTTP headers from the server response: At left, when the payload comes in JSON format, and at right, when the payload comes in binary format.

If I try to decompress it with brotli, I get:

# brotli -d outputfile.br 
corrupt input [outputfile.br]

Payload example: outputfile.gz

% file outputfile 
outputfile: Applesoft BASIC program data, first line number 15

Process

n/a

Current and expected result

Expected results: Payload always in JSON format.

Current results: Binary payload received in an arbitrary way.

Improve documentation to explain parameters

Summary

Q5. Can we document expected value descriptions for each querystring param:
It is hard from the doc API to see example values or intent of each querystring parameter. For example, where should I be looking at what type of data is expected for "q" to cves.json route?

CVEs endpoint:

q: any string - pattern checks for CVEs that have cve.description or cve.ubuntu_description like it. (I should make it check for cve.id too)
priority: can be one of the following: unknown, negligible, low, medium, high, critical
package_name: package name
limit: integer - defaults to 20
offeset: integer - defaults to 0
component: can be one of the following: main, universe
version: can be one of the releases_codenames form the file (you need the same number of versions and statuses values)
status: can be one of these: released, DNE, needed, not-affected, deferred, needs-triage, ignored, pending (you need the same number of versions and statuses values)
order_by: “oldest” or leave empty for newest first (just realised it does not work right now for CVE)

USNs endpoint:

details: any string - pattern checks and selects notices that have notice.id or notice.details or any notice.cves.id like it
release: can be one of the releases_codenames form the file (you need the same number of versions and statuses values)
limit: integer - defaults to 20
offeset: integer - defaults to 0
order_by: “oldest” or leave empty for newest first

Update releases endpoint

notices.json "details" parameter not returning matches known related CVEs.

Summary

Hidden USNs are being shown by default on public CVE queries. Not certain if USNs should remain hidden from CVE queries by default unless an additional parameter is provided. Also when trying to search from notices.json API providing the same CVE in the "details" parameter, no matches are found with or without hidden params.

Process

curl -X GET "https://staging.ubuntu.com/security/cves/CVE-2017-9233.json" -H "accept: application/json"
...
"notices": [                                                                    
   "USN-6100-1"                                                                 
]  

# No USNs returned from notices.json API route when searching for existing CVE with and without the hidden param
curl -X GET "https://staging.ubuntu.com/security/notices.json?details=CVE-2017-9233" -H "accept: application/json"
{"limit":20,"notices":[],"offset":0,"total_results":0}

# In production, related CVEs can be searched from notices.json API route using "details" param:
curl -X GET "https://ubuntu.com/security/notices.json?details=CVE-2017-9233" -H "accept: application/json"
{ USN-3356-1 ... USN-3356-2... "total_results":2}

Current and expected result

Current results were empty when querying security/notices.json for a known CVE related to a hidden USN in either the public query or the extra hidden param query.

Results from notices.json?details=CVE-id should contain any published USN related to CVE-id without any hidden USNs.
When the extra hidden param is provided to the above query, both published and hidden USNs should be returned.

Add new field `channel`

The security team would like to request that a new (optional) field be added into the NoticePackage group, and have it be called channel. The idea is that field will be related to a string/text argument.

We request that it be optional because it will most likely not be used for USNs, but this new field will be necessary for the recently created SSNs.

No related "cves" metadata on USN-4754-2

Summary

In order to properly fix a USN, ua fix USN-4754-2 attempts to find any additional USNs by querying the related "cves" returned for the USN response object.

In the case of USN-4754-2 there are no related "cves" (probably because f USN-4754-2 introduced a regression). In this case, there is not enough information or breadcrumbs for the client to discover other USNs which could subsequently fix the related issue. Do we know why the original related CVE is stricken from this metadata? Is this a class of USNs that we can better handle to allow discoverability of related USNs or CVEs?

security/release endpoints are inconsistent with `support_tag` key

For the /security/release endpoints, when perfoming a GET on a specific release there is a support_tag field (as seen in the swagger doc; for the current active releases this field contains either the empty string, "LTS", or "ESM", depending on the release. However, I don't see in the swagger documentation a way to add or update this field; it's not mentioned in the swagger API doc for:

The implementation does indeed match the swagger docs, as you can see through queries:

$ curl -s -X GET "https://ubuntu.com/security/releases/focal.json" -H "accept: application/json" | jq .
{
  "codename": "focal",
  "development": false,
  "esm_expires": "2030-04-30T00:00:00",
  "lts": true,
  "name": "Focal Fossa",
  "release_date": "2020-04-23T00:00:00",
  "support_expires": "2025-04-30T00:00:00",
  "support_tag": "LTS",
  "version": "20.04"
}

and attempts to make a no-change update that includes all fields:

# invoking this script with --debug only reports the json to submit and the url it would submit to
$ ./scripts/post-release-to-web-cve-tracker.py --action update focal --debug
PUT https://ubuntu.com/security/releases/focal.json
{
  "name": "Focal Fossa",
  "version": "20.04",
  "codename": "focal",
  "development": false,
  "lts": true,
  "support_tag": "LTS",
  "release_date": "2020-04-23T00:00:00",
  "support_expires": "2025-04-30T00:00:00",
  "esm_expires": "2030-04-30T00:00:00"
}

# the attempt to update the endpoint, which fails
$ ./scripts/post-release-to-web-cve-tracker.py --action update focal 
focal: <Response [422]> {
  "errors": "{'json': {'support_tag': ['Unknown field.']}}",
  "message": "Invalid payload"
}

The endpoint will correctly be updated if the support_tag key is not included. Is this correct behavior? When it comes time to add 22.04/impish+1, which team is responsible for adding the "LTS" support_tag to the release? Similarly, in April of 2023, what will the process be for changing bionic/18.04's support_tag to "ESM"?

Thanks.

Updating hidden notice returns 500

Make landing page a redirect

Summary

Make api homepage redirect to https://ubuntu.com/security/notices. Just as the current https://securitu.ubunut.com behaves.

Investigate timeouts

We are getting random timeouts for the GET endpoints. Typically the problem resolves itself after a few tries/ some time, but we obviously still need to find the reason.

Different results for same CVE from different endpoints

Summary

When querying the same CVE in /security/cves.json and in /security/cves/{cve_id}.json different results are returned in the packages list.
When querying the same CVE in /security/cves.json and in /security/cves/{cve_id}.json, results are returned only from the second API endpoint.

Process

Example for 1: When querying CVE-2018-4323 in /security/cves.json, the response has only one package (webkitgtk) in packages.
When querying the same CVE in /security/cves/CVE-2018-4323.json the response has 5 packages.

Example for 2: When querying CVE-2021-38191 in /security/cves.json there are no results, but in /security/cves/CVE-2021-38191.json there are results returned.

If I'm not wrong, I think the problem (at least for 1) is here: https://github.com/canonical-web-and-design/ubuntu-com-security-api/blob/master/webapp/views.py#L211
(statuses overwritten in each iteration)

Current and expected result

The expected result is that the API endpoints will return the same results for a specific CVE, currently they are returning different results.

Thank you! (Hadas from the Snyk Security team)

Investigate if we can shrink the cves.json and notices.json endpoint payloads

Summary

At the moment we are sending detailed payloads of each CVE and Notice. Perhaps sending the detailed data should be done just in the single CVE/Notice endpoint, while the get multiple CVE/Notice endpoints keeps everything general.

This should improve the performance of those endpoints too, as we wouldn't have to load as much information in the memory.

We'd have to discuss this with Security and UA-fix team, to make sure the approve it.

Groovy marked as `devel` on releases API

Summary

When requesting the details on the Groovy 20.10 release from the releases API the release is marked as development=true.

Process

curl -X GET "https://ubuntu.com/security/releases/groovy.json" -H "accept: application/json"
Results state groovy as in development

Current and expected result

Groovy to be marked as EOL / development=false.

Screenshot

Update run image command

When trying to debug why this command is failing, it would be useful to see the server output

API docs not accessible in Firefox, Chrome because of invalid content-type

The API docs cannot be viewed or rendered in Firefox/Linux, Chrome/Linux, Firefox/macOS nor Safari/macOS because of the invalid content-type for one of the external assets linked on the page:

<title>Security API Docs</title>
  <script src="[https://unpkg.com/[email protected]/swagger-ui-bundle.js](view-source:https://unpkg.com/[email protected]/swagger-ui-bundle.js)" crossorigin></script>
  <script src="[https://unpkg.com/[email protected]/swagger-ui-standalone-preset.js](view-source:https://unpkg.com/[email protected]/swagger-ui-standalone-preset.js)" crossorigin></script>

$ curl -Isq https://unpkg.com/[email protected]/swagger-ui-standalone-preset.js | grep content
content-type: application/javascript; charset=utf-8

vs.

$ curl -Isq https://unpkg.com/[email protected]/swagger-ui-bundle.js | grep content-type
content-type: text/html; charset=utf-8

Which results in the following:

Oddly, this does work using TorBrowser on both Linux and macOS.

[Feature request] Add the option to query for multiple CVEs in the query, possibly by specifying multiple ?q=CVE-ID1&q=CVE-ID2

Summary

We would like to have the option to query in a single query multiple CVEs (e.g. 10, 20, etc) so we don't have to query the API n-times.

Process

N/A

Current and expected result

Be able to query multiple CVEs.

Screenshot

N/A

Browser details

N/A

Missing CVEs

Original title: CVE API query [ https://ubuntu.com/security/cves.json ] on package "apport" with limit=5 and offset=5 for pagination, is not fully reliable, some CVEs might by omitted by the API server

Summary

According to Ubuntu Security CVE website [https://ubuntu.com/security/cve] there are 38 CVEs listed for package "apport", Status=Released, UbuntuVersion=Xenial
https://ubuntu.com/security/cve?package=apport&version=xenial&status=released

Process

However when I try to fetch the list through CVE API query [ https://ubuntu.com/security/cves.json ] with batches of 5, and stepping with offset=5, then it can happen that some CVEs are omitted by the API server
like "CVE-2021-32556"

for i in $(seq 0 300); do curl -s -X GET -H "Content-Type: application/json" "https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&limit=5&offset=$((${i}*5))" | python -m json.tool | grep "^ "id|^ "published"; sleep 1; done

Currently the workaround we try use, and seems it works, is that we decrease the "offset" with 1, so that means limit=5 and offset=4, and then eliminate duplicate CVEs (due to the smaller offset) then we observe that previously omitted CVE is now reported by the CVE API server, so now we also receive the "CVE-2021-32556" , which was previously missing when we used the correct offset of 5 (limit of 5)

for i in $(seq 0 300); do curl -s -X GET -H "Content-Type: application/json" "https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&limit=5&offset=$((${i}*4))" | python -m json.tool | grep "^ "id"; sleep 2; done

new update (2021.11.19): customer reported that even the upmentioned workaround does not provide reliable results, even though for myself it seemed it worked, but based on the latest report from customer, it looks like even this workaround is not reliable at all

Please let us know when the fix will be ready, currently my customer is falling back on the CVE website query, however they need the CVE API-based query to be fixed as soon as possible.

Change endpoints to end with .json

Summary

Make cve and usn endpoints end with .json.

Incorrect CVE status for 16.04 ESM

Summary

Currently the web view shows for say https://ubuntu.com/security/CVE-2020-15180 that the status for say mariadb-10.0 for 16.04 ESM is 'Ignored (end of standard support, was needs-triage)' - this is incorrect, as can be seen in the CVE tracker git repo at https://git.launchpad.net/ubuntu-cve-tracker/tree/active/CVE-2020-15180#n25 - as that is the status for 16.04 LTS - and there is no current status for 16.04 ESM. The security team plans to add an additional field to each CVE to track the status of CVEs for 16.04 ESM and so once this is done this should be used instead.

Similarly, for the recently announced exim4 CVEs, taking https://ubuntu.com/security/CVE-2020-28024 as an example, this shows ignored for 16.04 ESM - and this is creating a lot of confusion amongst customers, as the real status for this CVE is needs-triage (https://git.launchpad.net/ubuntu-cve-tracker/tree/active/CVE-2020-28024#n26)

Process

Compare web view to the CVE tracker git.

Current and expected result

The status should match that which is in the CVE files in the git CVE tracker.

504 when requesting 100 notices

History here: https://chat.canonical.com/canonical/pl/4m1kkwir1f8x3fabk9b4dtgyjy

At the time of writing, https://ubuntu.com/security/notices.json works, https://ubuntu.com/security/notices.json?limit=20 works, but https://ubuntu.com/security/notices.json?limit=100 produces a 504 error. It's not clear right now if this has been the case for a long time or is a recent development.

Requesting 100 notices is probably not an unreasonable ask. We should see if it's possible to provide this. If it's not possible we should make it clear the maximum number that's supported by the API for querying at any one time.

Update security api to reflect changes made to ubuntu.com db

Summary

I made notice search queries case insensitive and added the ability to search by title in this PR on the ubuntu.com repo. Those same changes to the query filter need to be pushed to the security api repo as well.

Original issues for reference:
canonical/ubuntu.com#9454
canonical/ubuntu.com#9856

Test related_notices

Make sure we add a test for the work in #80.

Change API to return better error messages

Summary

Q6. Desire more concrete feedback from API on inability to find matches:
If I pass an invalid package name to cves.json?package=bogus I get an empty list response with no feedback that I have possible sent an "unknown package name". Is there anything we can do to inform a client about "unknown package name"? This avoids cases where someone (ua client) provides incorrect packages=cloud-int on a package that may have existing CVEs. We don't want to report None to them by accident.

{
"cves" : [],
"limit" : 20,
"offset" : 0,
"total_results" : 0
}

I'm not sure how best to provide more informative feedback, maybe a key for errors in the response that contains a list of messages/warnings? Or HTTP code 404 with "message" details? I want an API to be noisy if I've bungled something.

Make API publicly available 3rd try

RT link: https://rt.admin.canonical.com/Ticket/Display.html?id=128604&results=608d25c102559890d8074a12a5cfa24e

Create `is_hidden` filter for USN

Optimise /notices.json

Summary

/notices.json takes a long time to load.

intermittent timeouts or Gateway timeout errors from Security API routes for specific cves/{cve_id}.json or notices/{usn_id}.json

Summary

[Please describe the issue.]
From a remote client perspective, we are seeing fairly frequent timeouts either (Gateway timeout error 504) of just a 30 second lag in responses from the Security API when performing GETs via either curl or python requests to either of the routes u.com/security/cves/{cve_id}.json or u.com/security/notices/{notice_id}.json.

Process

repeated curl attempts over a period of time seem to hit this issue (pointing potentially to an HA Proxy round-robing config type issue maybe)?

curl -X GET "https://ubuntu.com/security/notices.json?details=USN-4510-1" -H "accept: application/json"
 curl https://ubuntu.com/security/notices/USN-4559-1.json -H "accept: application/json" > usn.3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100  1444  100  1444    0     0     46      0  0:00:31  0:00:30  0:00:01   338

The network timeouts talking to the security API seem to hit our integration tests failly frequently too (which leverages python requests for GETs against the Security API : canonical/ubuntu-pro-client#1374

Current and expected result

It seems strange to have a 30 second timeout on repeated requests to the Security API. Is there something either misconfigured with an HA frontend or is the a per-client throttling limit that we are hitting when we try 3-5 requests back to back against the API that is reducing out throughput of API calls?

Browser details

N/A

The "cves.json" endpoint doesn't provide all the vulnerabilities

Summary

There are certain vulnerabilities that can be obtained individually but aren't returned by the general API using a filter.
See example below.

Process

This CVE can be obtained individually

https://ubuntu.com/security/cves/CVE-2020-1951.json

{"bugs":["http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=954302"],"cvss3":5.5,"description":"\nA carefully crafted or corrupt PSD file can cause an infinite loop in\nApache Tika's PSDParser in versions 1.0-1.23.","id":"CVE-2020-1951","mitigation":"","notes":[],"notices":[{"cves_ids":["CVE-2020-1951","CVE-2020-1950"],"description":"It was discovered that Apache Tika can have an excessive memory usage by\nusing a crafted or corrupt PSD file. An attacker could use it to cause a\ndenial of service (crash). (CVE-2020-1950, CVE-2020-1951)\n","id":"USN-4564-1","instructions":"In general, a standard system update will make all the necessary changes.\n","is_hidden":false,"published":"2020-10-05T17:29:37.430543","references":[],"release_packages":{"xenial":[{"description":"A content analysis toolkit","is_source":true,"name":"tika","version":"1.5-4ubuntu0.1"},{"is_source":false,"is_visible":true,"name":"libtika-java","pocket":"security","source_link":"https://launchpad.net/ubuntu/+source/tika","version":"1.5-4ubuntu0.1","version_link":"https://launchpad.net/ubuntu/+source/tika/1.5-4ubuntu0.1"}]},"summary":"Apache Tika could be made to crash if it opened a specially crafted\nfile.\n","title":"Apache Tika vulnerabilities","type":"USN"}],"notices_ids":["USN-4564-1"],"packages":[{"debian":"https://tracker.debian.org/pkg/tika","name":"tika","source":"https://ubuntu.com/security/cve?package=tika","statuses":[{"component":null,"description":"","pocket":null,"release_codename":"bionic","status":"needs-triage"},{"component":null,"description":"reached end-of-life","pocket":null,"release_codename":"eoan","status":"ignored"},{"component":null,"description":"","pocket":null,"release_codename":"focal","status":"needs-triage"},{"component":null,"description":"reached end-of-life","pocket":null,"release_codename":"groovy","status":"ignored"},{"component":null,"description":"reached end-of-life","pocket":null,"release_codename":"hirsute","status":"ignored"},{"component":null,"description":"reached end-of-life","pocket":null,"release_codename":"impish","status":"ignored"},{"component":null,"description":"","pocket":null,"release_codename":"jammy","status":"needs-triage"},{"component":null,"description":"","pocket":null,"release_codename":"precise","status":"DNE"},{"component":null,"description":"","pocket":null,"release_codename":"trusty","status":"DNE"},{"component":null,"description":"","pocket":null,"release_codename":"upstream","status":"needs-triage"},{"component":null,"description":"1.5-4ubuntu0.1","pocket":null,"release_codename":"xenial","status":"released"}],"ubuntu":"https://packages.ubuntu.com/search?suite=all&section=all&arch=any&searchon=sourcenames&keywords=tika"}],"patches":{"tika":[]},"priority":"low","published":"2020-03-23T14:15:00","references":["https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-1951","https://www.openwall.com/lists/oss-security/2020/03/18/4","https://lists.apache.org/thread.html/rd8c1b42bd0e31870d804890b3f00b13d837c528f7ebaf77031323172%40%3Cdev.tika.apache.org%3E","https://lists.debian.org/debian-lts-announce/2020/03/msg00035.html","https://ubuntu.com/security/notices/USN-4564-1","https://ubuntu.com/security/notices/USN-4564-1"],"status":"active","tags":{"tika":[]},"ubuntu_description":""}

And also in the cves.json endpoint using a filter

https://ubuntu.com/security/cves.json?limit=1&q=CVE-2020-1951

{"cves":[{"bugs":["http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=954302"],"cvss3":5.5,"description":"\nA carefully crafted or corrupt PSD file can cause an infinite loop in\nApache Tika's PSDParser in versions 1.0-1.23.","id":"CVE-2020-1951","mitigation":"","notes":[],"notices":[{"cves_ids":["CVE-2020-1951","CVE-2020-1950"],"description":"It was discovered that Apache Tika can have an excessive memory usage by\nusing a crafted or corrupt PSD file. An attacker could use it to cause a\ndenial of service (crash). (CVE-2020-1950, CVE-2020-1951)\n","id":"USN-4564-1","instructions":"In general, a standard system update will make all the necessary changes.\n","is_hidden":false,"published":"2020-10-05T17:29:37.430543","references":[],"release_packages":{"xenial":[{"description":"A content analysis toolkit","is_source":true,"name":"tika","version":"1.5-4ubuntu0.1"},{"is_source":false,"is_visible":true,"name":"libtika-java","pocket":"security","source_link":"https://launchpad.net/ubuntu/+source/tika","version":"1.5-4ubuntu0.1","version_link":"https://launchpad.net/ubuntu/+source/tika/1.5-4ubuntu0.1"}]},"summary":"Apache Tika could be made to crash if it opened a specially crafted\nfile.\n","title":"Apache Tika vulnerabilities","type":"USN"}],"notices_ids":["USN-4564-1"],"packages":[{"debian":"https://tracker.debian.org/pkg/tika","name":"tika","source":"https://ubuntu.com/security/cve?package=tika","statuses":[{"component":null,"description":"","pocket":null,"release_codename":"bionic","status":"needs-triage"},{"component":null,"description":"reached end-of-life","pocket":null,"release_codename":"eoan","status":"ignored"},{"component":null,"description":"","pocket":null,"release_codename":"focal","status":"needs-triage"},{"component":null,"description":"reached end-of-life","pocket":null,"release_codename":"groovy","status":"ignored"},{"component":null,"description":"reached end-of-life","pocket":null,"release_codename":"hirsute","status":"ignored"},{"component":null,"description":"reached end-of-life","pocket":null,"release_codename":"impish","status":"ignored"},{"component":null,"description":"","pocket":null,"release_codename":"jammy","status":"needs-triage"},{"component":null,"description":"","pocket":null,"release_codename":"precise","status":"DNE"},{"component":null,"description":"","pocket":null,"release_codename":"trusty","status":"DNE"},{"component":null,"description":"","pocket":null,"release_codename":"upstream","status":"needs-triage"},{"component":null,"description":"1.5-4ubuntu0.1","pocket":null,"release_codename":"xenial","status":"released"}],"ubuntu":"https://packages.ubuntu.com/search?suite=all&section=all&arch=any&searchon=sourcenames&keywords=tika"}],"patches":{"tika":[]},"priority":"low","published":"2020-03-23T14:15:00","references":["https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-1951","https://www.openwall.com/lists/oss-security/2020/03/18/4","https://lists.apache.org/thread.html/rd8c1b42bd0e31870d804890b3f00b13d837c528f7ebaf77031323172%40%3Cdev.tika.apache.org%3E","https://lists.debian.org/debian-lts-announce/2020/03/msg00035.html","https://ubuntu.com/security/notices/USN-4564-1","https://ubuntu.com/security/notices/USN-4564-1"],"status":"active","tags":{"tika":[]},"ubuntu_description":""}],"limit":1,"offset":0,"total_results":1}

But with another CVE we have a different behavior.
This CVE can be obtained individually

https://ubuntu.com/security/cves/CVE-2021-1345.json

{"bugs":null,"cvss3":null,"description":null,"id":"CVE-2021-1345","mitigation":null,"notes":[{"author":"ubuntu-security","note":"Does not apply to software found in Ubuntu."}],"notices":[],"notices_ids":[],"packages":[],"patches":null,"priority":null,"published":null,"references":["https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-1345"],"status":"not-in-ubuntu","tags":null,"ubuntu_description":null}

But not using a filter

https://ubuntu.com/security/cves.json?limit=1&q=CVE-2021-1345

{"cves":[],"limit":1,"offset":0,"total_results":0}

Current and expected result

It seems that the cves.json endpoint doesn't contain all the available vulnerabilities.

This endpoint should have the same content than the cves/{cve_id}.json one.

Browser details

Brave browser: [Versión 1.43.89 Chromium: 105.0.5195.102 (Build oficial) (64 bits)]

Fix swagger documentation

Summary

We are using an old swagger documentation version. We have to update it.

Add endpoints for CVE and Releases

Summary

Add CVE endpoints for bulk import and delete.
Add release endpoints for create and delete.

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined