gastamper / annparse Goto Github PK

Parser that generates lists from advisories posted to the CentOS/CentOS-CR announce mailing lists for easier updating of systems

Rust 100.00%

advisories centos yum

annparse's People

Watchers

annparse's Issues

Only load cache files from possible years

No point checking 2016 for a 2021 advisory.
Ensure check year +/-1 to cover late CVEs.

Support scanning adjacent years if no hits

Late December advisories could be addressed in the following year's batch of updates.

Fold -c functionality into basic functionality

As title states.

Mailing list archives may not be compressed

f.e. September CR-announce list is .txt not .txt.gz as of 10/2/19.
http://archive.is/RBZwJ

Handle Error in unwrap for get_archive_list macro

SSL certificate error raised pulling CESA-2021:3438 today. Needs a check in get_archive_list! macro's unwrap in case of error.

Decide how to handle "wrong-year" advisories

Examples:
CESA-2019:2003 (April 2020 CR)
CEBA-2018:0748 (October 2020 CR)

Worth bothering with? If so, how? Would have to undo neighbor-year checking of cache, which otherwise makes things a lot faster.

Also these advisories don't show in online mode since they are in the "incorrect" year (2018 in 2020). Only option would be to iterate over other years when no hits found in current & adjacent year, and that seems messy and spammy.

Duplicate build_line insertions in online mode

[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: 48e16b680425854202c565c6c1020cafea961171270dac3c3c3c2ca9aa44c7da  ghostscript-9.07-31.el7_6.11.x86_64.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: 4dc03be1378cbddfeafaa92807055c10492f29d29eae52e1bc618fa13c80ef74  ghostscript-cups-9.07-31.el7_6.11.x86_64.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: b425f4a3242e5fded833e0289d9422b5298518abed26edab428f82152fdbc57a  ghostscript-devel-9.07-31.el7_6.11.i686.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: 9b4f91aae4030ce6a3feac301f64a589b33722a6ce39770f67c588fce502cfb3  ghostscript-devel-9.07-31.el7_6.11.x86_64.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: d33a119106be2ed4b9078ad589ba4c77567d098d801b5ecff16b16b250ff1db4  ghostscript-doc-9.07-31.el7_6.11.noarch.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: f3ce22804cae204cc18a4ff3ea78ed277a4cfa73c4044289e0740f5c506da709  ghostscript-gtk-9.07-31.el7_6.11.x86_64.rpm
[2021-07-02T13:52:55Z DEBUG annparse] Advisory matched: CESA-2019:1017, Important CentOS 7 ghostscript
[2021-07-02T13:52:55Z TRACE annparse] build_line: {
        "ghostscript-devel-9.07-31.el7_6.11.x86_64.rpm",
        "ghostscript-9.07-31.el7_6.11.i686.rpm",
        "ghostscript-devel-9.07-31.el7_6.11.i686.rpm",
        "ghostscript-9.07-31.el7_6.11.x86_64.rpm",
        "ghostscript-doc-9.07-31.el7_6.11.noarch.rpm",
        "ghostscript-gtk-9.07-31.el7_6.11.x86_64.rpm",
        "ghostscript-cups-9.07-31.el7_6.11.x86_64.rpm",
    }
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-devel
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-devel
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-doc
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-gtk
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-cups
[2021-07-02T13:52:55Z DEBUG annparse] Final package list: ghostscript ghostscript-devel ghostscript-gtk ghostscript-doc ghostscript-cups
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CEBA-2019:1015
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1022
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CEEA-2019:1212
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CEBA-2019:1227
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CEBA-2019:1226
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1181
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1180
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1169
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1131
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CEEA-2019:1210
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1178
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1177
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1168
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1235
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1228
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1265
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1267
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1264
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1024
[2021-07-02T13:52:55Z TRACE annparse] Advisory found: CESA-2019:1017
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: 758ec53f84bc80379756e27eb041f90feeede7c1705ef517b099f9f577f625de  ghostscript-9.07-31.el7_6.11.i686.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: 48e16b680425854202c565c6c1020cafea961171270dac3c3c3c2ca9aa44c7da  ghostscript-9.07-31.el7_6.11.x86_64.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: 4dc03be1378cbddfeafaa92807055c10492f29d29eae52e1bc618fa13c80ef74  ghostscript-cups-9.07-31.el7_6.11.x86_64.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: b425f4a3242e5fded833e0289d9422b5298518abed26edab428f82152fdbc57a  ghostscript-devel-9.07-31.el7_6.11.i686.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: 9b4f91aae4030ce6a3feac301f64a589b33722a6ce39770f67c588fce502cfb3  ghostscript-devel-9.07-31.el7_6.11.x86_64.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: d33a119106be2ed4b9078ad589ba4c77567d098d801b5ecff16b16b250ff1db4  ghostscript-doc-9.07-31.el7_6.11.noarch.rpm
[2021-07-02T13:52:55Z TRACE annparse] split_sort: Inserting: f3ce22804cae204cc18a4ff3ea78ed277a4cfa73c4044289e0740f5c506da709  ghostscript-gtk-9.07-31.el7_6.11.x86_64.rpm
[2021-07-02T13:52:55Z DEBUG annparse] Advisory matched: CESA-2019:1017, Important CentOS 7 ghostscript
[2021-07-02T13:52:55Z TRACE annparse] build_line: {
        "ghostscript-cups-9.07-31.el7_6.11.x86_64.rpm",
        "ghostscript-gtk-9.07-31.el7_6.11.x86_64.rpm",
        "ghostscript-9.07-31.el7_6.11.x86_64.rpm",
        "ghostscript-devel-9.07-31.el7_6.11.x86_64.rpm",
        "ghostscript-9.07-31.el7_6.11.i686.rpm",
        "ghostscript-devel-9.07-31.el7_6.11.i686.rpm",
        "ghostscript-doc-9.07-31.el7_6.11.noarch.rpm",
    }
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-cups
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-gtk
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-devel
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-devel
[2021-07-02T13:52:55Z DEBUG annparse] Got package: ghostscript-doc
[2021-07-02T13:52:55Z DEBUG annparse] Final package list: ghostscript ghostscript-cups ghostscript-gtk ghostscript-devel ghostscript-doc

CESA-2020:0124 returns nothing

CESA-2020:0124 returns no packages on test system

Allow using local list cache

We should be good users and actually build a local cache (because previous months never(?) change) and use that instead of hammering the mailing list archive website every time the program is run. Will also greatly speed execution.

cache changes made --offline mandatory?

target/debug/annparse -a CESA-2021:4353
error: The following required arguments were not provided:
    --offline

USAGE:
    annparse --advisory <advisory> --cache_path <cache_path> --offline

For more information try --help

Neighbor year checking not supported in online mode

Seems a bit of an oversight.

Requires manually passing year

This should be parsed from advisory name.

fs::read should be match'd on its own to catch errors there

annparse/src/main.rs

Line 225 in d02cf01

// TODO: fs::read should be match'd on its own to catch errors there

Check CR by default or only with flag?

Worth considering which situation is preferable:

Current state - checking CR requires manually passing a flag
Default to check CR
2a. ... only if no hits in CentOS-announce?
2b. ... before or after CentOS-announce?

For our purposes 2b/before is probably preferable.

[2021-07-02T13:52:52Z TRACE annparse] Decompressed 179072 bytes
[2021-07-02T13:52:52Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-November.txt.gz
[2021-07-02T13:52:52Z TRACE annparse] Decompressed 127123 bytes
[2021-07-02T13:52:52Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-October.txt.gz
[2021-07-02T13:52:52Z TRACE annparse] Decompressed 238787 bytes
[2021-07-02T13:52:52Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-September.txt.gz
[2021-07-02T13:52:52Z TRACE annparse] Decompressed 327692 bytes
[2021-07-02T13:52:53Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-August.txt.gz
[2021-07-02T13:52:53Z TRACE annparse] Decompressed 29627 bytes
[2021-07-02T13:52:53Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-July.txt.gz
[2021-07-02T13:52:53Z TRACE annparse] Decompressed 296886 bytes
[2021-07-02T13:52:53Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-June.txt.gz
[2021-07-02T13:52:53Z TRACE annparse] Decompressed 128458 bytes
[2021-07-02T13:52:53Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-May.txt.gz
[2021-07-02T13:52:53Z TRACE annparse] Decompressed 121794 bytes
[2021-07-02T13:52:54Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-April.txt.gz
[2021-07-02T13:52:54Z TRACE annparse] Decompressed 270072 bytes
[2021-07-02T13:52:54Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-March.txt.gz
[2021-07-02T13:52:54Z TRACE annparse] Decompressed 243070 bytes
[2021-07-02T13:52:54Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-February.txt.gz
[2021-07-02T13:52:54Z TRACE annparse] Decompressed 303596 bytes
[2021-07-02T13:52:55Z DEBUG annparse] Status 200 OK for https://lists.centos.org/pipermail/centos-announce/2019-January.txt.gz
[2021-07-02T13:52:55Z TRACE annparse] Decompressed 63447 bytes
[2021-07-02T13:52:55Z TRACE annparse] Archive bundle found, length 179072

Matches first hit (December).

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble