ps2alerts / aggregator Goto Github PK

Websocket client which subscribes and processes data from the Census Streaming Service, passing it to the PS2Alerts API via RabbitMQ.

Home Page: https://ps2alerts.com

License: GNU General Public License v3.0

JavaScript 2.38% Dockerfile 0.15% Shell 0.32% TypeScript 97.15%

planetside2 typescript websocket nodejs kubernetes ansible terraform census rabbitmq ioc

aggregator's Introduction

PS2Alerts Aggregator

The aggregator collection script that powers PS2Alerts.com.

If you wish to contribute, please join our Discord located at: https://discord.gg/7xF65ap

Preface

This project powers the PS2Alerts website. Its primary purpose is to act as a Data Collector, which listens in on events coming in from the Census Stremaing Service and formats that data into a legible format, commits it to a database, which in turn the API will serve to the frontend website.

Installation

Repo submodules

NOTE: After you have cloned this repository, you also need to install the git submodule which contain various constants used across the PS2Alerts repositories. Simply execute

git submodule update --init --recursive

If in the future your application is performing weird saying it can't find references to things, pull the latest modules in via running ./module-update.sh.

Installing and running the app

For first time runs, you must run ps2alerts-aggregator-init, which goes ahead and builds the base image required for the dev Docker image to run.

Run ps2alerts-start to start all associated services including this module. Dependencies will be handled via the bootstrap process.

Before first running the aggregator, you need to copy .env.example to .env and fill in your census service ID, as well as any other dependencies e.g. remote redis etc.

To start the aggregator for development, run ps2alerts-aggregator-dev. This will bootstrap the container with ENV vars etc and tail the docker logs, as you would if you ran it manually via NPM.

Workarounds

On Mac OS X, gyp has a hard dependency on XCode. If you encounter the below error, you will need to install XCode :-/

 Error: `gyp` failed with exit code: 1

Local dependencies

For local development, you're recommended to have the following installed:

Node v16+ recommend using NVM (Node Version Manager)
NPM

Contributions

Please check the issues list for where you can contribute to this project. For more information, click on the Discord link above and have a chat with the developers.

Points of note

If you don't quite understand IoC, I suggest you create an application as per the Inversify tutorials, hopefully it'll click. Feel free to ask any of the collaborators for help.

File structure

`/src`

All application code is located within /src.

All provisioning and supported services are located within /provisioning. This includes the development environment, staging, and production build methods.

All pipelines are located within ./github/workflows, which performs consistency tests and checks.

`/src/bootstrap.ts`

This is where we instantiate the IoC container and load the Kernel modules.`

`/src/index.ts`

This is where the fun begins. Index.ts loads the Kernel, which in turn loads the Container, which in turn sets everything up, and then once that's all running, listens for kernel level exceptions which we haven't caught within the application and logs it, then gracefully terminates the application.

`/src/authorities`

This folder contains various subroutines which checks various things, e.g. shutting overdue alerts, population statistics gathering etc.

`/src/bootstrap`

Kernel.ts - herein lies the Kernel, essentially the container for the application. This boots and loads all services and is where everything begins.

`/src/config`

Herein contains all the application config information, some of it hardcoded, some of it from env vars.

`/src/constants`

This folder contains all of our enumerates and static data / game data which is constant.

`/src/data`

This is where our custom classes go, e.g. CharacterPresenceData.

`/src/drivers`

Contains drivers which provide override functionality, e.g. CensusCacheDriver which implements Redis on behalf of the Census Package.

`/src/exceptions`

Where our custom exceptions will exist. Currently has ApplicationException which provides a standard format.

`/src/factories`

Contains various factories which create instantiated classes based on context. For example, Territory Calculator is finite, and this factory creates a new class for each scenario.

`/src/handlers`

Where the meat of the application will live. This is where all the event handlers will exist, e.g. DeathEvent. This is where all the processing, database updates, event emits etc will be triggered. This folder will get quite large eventually.

It also contains useful services such as CharacterBroker, who's job is to go to Census and retrieve character information.

`/src/instances`

Everything in PS2Alerts is driven around an instance. If there is no instance, it won't get recorded. Therefore, we have created the system to be able to define custom instance types, e.g. a Planetside Battles event on Jaeger. By default, we use the PS2AlertsMetagameInstabce. Each instance must implement the contained interface, which describes common attributes.

`/src/interfaces`

This is where our code interfaces will live. E.g. each Handler will have an associated parent Interface which each handler must adhere to. We will self-enforce usage of interfaces as it's simply good coding practice.

`/src/logger`

Where the logging class exists. May move into a service instead, but the concept of a service for us isn't quite the same as what you may expect from say PHP services. It's more of a utility class.

`/src/models`

Herein lies all of our models which we use to interact with MongoDB. Each model instantiates from /src/services/mongo/index.ts, and within each model contains the collection structure, along with an interface which enforces certain data structure patterns.

`/src/services`

This is where our services exist which are used throughout the application, such as connecting to Planetside 2's Census Streaming Service, Mongo and Redis connectivity.

`/utils`

This is where utility classes / functions will live.

aggregator's People

Contributors

Stargazers

Watchers

Forkers

marci4 ryanjsims alanrps

aggregator's Issues

Integrate websocket server

The old websocket created a websocket server which enabled the PS2 Frontend / website to connect to it and receive data from our collection websocket.

The purpose of this was to provide additional events and data which is useful to the website not available in Census. For example, when an alert ends, we can send through the final scorings and final data to the frontend webclient for a final render. It will also update the "realtime monitor" on the website frontend to denote that an alert has ended, or has started etc.

It appears Express may be a good use for this, @Hailot has previously implmented it, but unsure whether it's still going to work with typescript.

Assigning myself and Hailot to this, so we can re-review and re-implement if required.

Decide on unit testing framework

We should decide on a unit testing framework.

Possibilities are

Jest
Mocha
Jasmine

Create direct connection to Census streaming service

Related to #3 , we need to subscribe to census directly for steaming updates. We will require the following:

Mechanism to programmatically subscribe to census, by server and ideally by zone
Reconnection to the socket when connections go stale or we've not received data when we should (I remember jhett having to do this)
Event subscription filtering
Downtime detection, to be transmitted to the frontend should the service be unavailable or Census static API not providing data

Write command to hydrate Weapons & Vehicles from the API

Now we have the ability to insert data, we need the ability to scrape Weapon data from Census and store it in the database for references.

Decide whether to scrap the websocket code and start again, or refactor

So it's no secret the websocket-server.js file is complete and utter trash when it comes to maintainability and readability.

It either needs completely nuking and starting again, or refactored into modules. I would prefer the latter as get the project working again then refactor later.

Thoughts?

Fix statics within MongoDBConnection

Will have a look tomorrow, but it is mostly about removing the static keywords so every new instance can have its own state.

Originally posted by @microwavekonijn in #46

ContainerModules are not executing "terminating" upon process termination

We need to investigate and test why this isn't occuring. Since we're using CI/CD methodologies and the fact the containers are meant to be treated like cattle, we need to make sure these calls are working.

Loosing connection to Census

For some currently unknown reason, we are loosing our connection to Census, leaving instances in invalid states and not updating.

For now until these issues have been resolved, I'm going to implement a check which confirms we have got any events from Census every 5 minutes, if not, restart the census connection.

Create EventMessage Handlers

This will be the meat of the websocket. These handlers will take the incoming message from census, parse it, perform any calculations, data storage, aggregation, emitting other events etc required in order to perform the correct logic for that event.

As of #16 , the foundation for the handlers have been created, as per the CensusProxy class.

Must have

Should have

GainExperience (for medic XP etc)
PlayerFacilityCapture
PlayerFacilityDefend
ContinentUnlock (currently a dead event in the API)

Could have

AchievementEarned
BattleRankUp

Don't care

ItemAdded
SkillAdded

Re-write the websocket into Typescript and Modules

~~It's been decided in #1 that we're going to refactor the application rather than totally gutting it in re-writing it.~~

We're converting the project into Typescript and writing the websocket from scratch.

It has also been agreed we should also convert this from normal JS to typescript, and use it as a standard for any further NodeJS work in the future.

Agree on eslint rules

Let's formulate a list of ESLint rules so we don't need to worry about it in PRs.

My personal suggestions other than what's already in:

EOF new line
New Lines after control structures (ifs, fors, eachs, switches etc)
New Lines after functions & constructs
New LInes after sets of properies (e.g. before construct)

Add FileTransport Logger transport

Need to be able to pipe the logs to a file, which could alternatively be ingested into a log scraper, also we can search locally for entries.

CensusStreamService is regarded as "started" but not actually connected

We currently have a race condition where the Kernel has regarded the app as fully started when actually Census is still connecting.

We need to add a promise of some sort to prevent the kernel marking the CensusStreamService as connected when we haven't yet received the ready event.

Create EditorConfig

We should have a editor config which should help the IDE follow the code guidelines in this project.

Attached you can find my config for now.
.editorconfig.txt

Create additional EventMessage Handlers

The required eventhandlers were added with #18.
This issue should cover all eventhandlers, which are not required in the short term

Should contain

Update ps2census to version 0.2.6

After microwavekonijn/ps2census#14 is fixed, we should update to the latest ps2census version containing the fix.

The following things have to be solved with this issue:

Re enable world validator check in CensusProxy.ts

Refactor logic to be more "instance" generic rather than Alert specific

Rather than tying the project directly into being Alert centric, there's no real reason why we can't make the project quite generic then add alert specific stuff on top of it.

This will then mean we can open the project up for more types of events, e.g. PSB ServerSmashes. Right now, it won't require much work at all to convert into a generic system.

Rename "ActiveAlertAuthority" to "ActiveInstanceAuthority"
Rename all "Alert" models to "Instance" models
Refactor app rather than inserting "alertId" everywhere, have it insert an "instanceId" and rename the metagame event "instance" ID to "cenusInstance".
Add a new field to the "ActiveAlert" / "ActiveInstance" model called instanceType, which will be an ENUM of currently two fields: CensusMetagameEventInstance [1] and ScheduledEventInstance [2]

Population Tracking

Need the ability to monitor the amount of online players and ideally map them to a zone.

This data then on a per-alert / event basis needs to be dumped to an aggregate table so on the website we can plot the approximate number (within 5 minutes) of players on a zone during the alert / event.

This also needs to include #31 .

Implement Event Handlers

Thanks to #55, we now have the foundation for adding and updating data from the database. We now therefore need to implement the event handlers and create models as we go.

We also may need to add statistics (e.g. count up Kills on an alert for example) - therefore this marrys very well with #26.

Handlers

Priority

ContinentLockEventHandler
ContinentUnlockEventHandler
DeathEventHandler
FacilityControlEventHandler
MetagameEventEventHandler (naming lol)
VehicleDestroyEventHandler (doesn't seem to exist! :O)

Secondary

PlayerFacilityCaptureEventHandler (named wrong in codebase)
PlayerFacilityDefendEventHandler (ditto)
PlayerLoginEventHandler
PlayerLogoutEventHandler

Meh

AchievementEarnedEventHandler
BattleRankUpEventHandler

Models to create

Alert

Aggregates

See #26

Misc

PlayerLogin
PlayerLogout

Statics

Statics in my head is data that cannot change via the game, or requires customer service intervention to do so.

Player
Outfit

Dynamics

Dynamics are values of data that can be changed at any time, e.g. an outfit's tag etc.

Player
Outfit

Handle scenario where continents unlock during monitoring

Currently the application throws an error when a continent "powers up" after being unstable or upon locking. This won't happen during an alert, however should the application be expanded to monitor across a zone for say a killfest event, we need to handle it.

2020-07-23T14:13:04.462Z | error | FacilityControlEventHandler >> Error parsing FacilityControlEvent: Unable to insert FacilityControlEvent into DB! Alert: 10-49785 - ValidationError: oldFaction: `0` is not a valid enum value for path `oldFaction`.
{
    "alert": {
        "alertId": "10-49785",
        "instanceId": 49785,
        "world": 10,
        "zone": 2
    },
    "facility": 4401,
    "timestamp": 1595513585,
    "durationHeld": 840,
    "oldFaction": 0,
    "newFaction": 3,
    "isDefence": false,
    "outfitCaptured": "0"
}

Integrate updates to upstream Census library (v0.3.6)

Need to update the census library and make any changes required for integration

Create background task / service to prune online players if not seen in a threshold time

Need a background task to prune the online players for the PS2 worlds. We also need to update handlers to keep this information fresh as possible, using a "last seen" unix timestamp. The task will run through the online players and check their last online times, and if it's over a certain threshhold (currently thinking 10 mins) remove them from being online.

Originally posted by @Maelstromeous in #23

Integrate Census Library v0.2.4 upstream changes

@microwavekonijn has done a lot of work solving a lot of headaches for us, time to integrate the library.

When PS2 servers are down, socket falls into reconnection loop, causing 100% CPU

https://i.imgur.com/JpMvxzV.png

Need to add a backoff timeout delay so it doesn't continiously attempt to reconnect constantly

Implement forced alert ending

Metagame End messages are not 100% reliable. We need to implement the ability to close overdue alerts.

Integrate database layer and @Hailot 's models into Typescript

Spinoff from #16 ,

We need to migrate the database models @Hailot has made into Typescript and integrate Sequlize.

Create models for all events recieved trough census websocket

Integrate Database connection

~~So we've had some discussion and for now it seems logical to use TypeORM. It saves a lot of headaches creating schemas etc and it allows us to use entities which we just save.~~

Changed to Mongoose

I've been assured that it doesn't have any massive performance implications, so for now I'm happy to use it. My personal experience with ORMs is that they're slow and comebersome, but that might be just a PHP land issue.

Refactor: Store interface symbol with interface

Right now if you have an interface with an associated symbol, and you delete the file with the interface the symbol will still exist. To prevent people pulling their hair out of frustration let's let the symbol move in to the file of the interface(I am sure they are ready for that kind of commitment).

Use cenus for map from metagame_event_id to zone

The endpoint http://census.daybreakgames.com/s:/get/ps2/metagame_event?c:limit=500 should be used to map a event_id to Zone

Remove 3rd party API integration

The websocket is heavily dependent on Jhett12321's middleman API. Unfortunately that project is no longer being maintained. Code referencing the project needs removing and a common interface creating.

Implement Redis Cache driver

In prelude to #106 , we will eventually need to cache information coming from census. To ensure we don't get the biggest ban hammer that probably any PS2 developer will ever seek to bestow upon our poor meere mortal souls, we should implement a caching layer.

Discord transport can be ratelimited

Add a queue so that the ratelimit for messages send to a discord channel is not exceeded. I believe it is 5 messages per 1 minute. Can also be an opportunity to send logs in bulk as they are embeds?

Create Statistic / Aggregate Handlers

As part of #18 we now have a solid foundation where messages are being handled and will then be proxied off for processing. In effect the collection and validation of messages is now done.

Next step is to process the data into a useful format for the Alerts Statiatics dataset itself.

In the old code, there were very crude implementations which enabled aggregation of statistics, e.g. for a player how many kills they got on a per-alert basis for example. This was done by simply incrementing a number in a database row. However, it was done in a very poor manner which meant any changes to the database schema were a royal ballache, as the implementation was basically a vast set of update queries, each being a special snowflake.

Therefore, we now need to create Aggregate Handlers which updates the appropiate sections of the data set when we receieve relevent events. This is required to reduce the vast amount of processing required to generate the statistics, due to the sheer volume of the data. E.g., when we get a PlayerDeath event, we need to do the following things:

Validate the message (done)
Create a record for that player for the particular alert
Create a record (if it doesn't already exist) for the player for their Global Statistics (will explain why it's done here shortly)
If records already exist, increment the statistic for that player.

There are a few things to consider with this approach:

Are we going to retain player weapon data? e.g. Maelstrome26 got 20 kills with the Lasher in Alert #101234
Are we wanting to aggregate an entire set of a player's statistics on a per-alert level or globally? Would be nice to do it on a per-alert level where in the frontend we can denote "Player Maelstrome26 killed 5 players in a Magrider this alert" etc.
Are we comfortable logging a lot of player-level statistics? It can be quite a lot of data.
I'm also not proposing we store every single event we have on a player. We must do aggregation, update the relevent stat then drop the excess data. We're talking potentially millions of events per alert otherwise, which will fill up our disks very soon.
Do we wish to have a certain timeframe where very detailed statistics e.g. player's weapons on a vehicle is wiped after a certain period to compress data sizes?

From memory, below are the aggregates that the old site used:

Alert level aggregates

Player specific

Outfit specific

AlertOutfitAggregate - this holds per-outfit per-alert stats, combat, facilitycontrol captures etc
- Model
- Implemented in EventHandler(s)

Potentially we could add AlertOutfitVehicle and AlertOutfitWeapon aggregate but debatable for the usage of this

Alert aggregates

Global level aggregates

Proposed new aggregates

The below was never in the original code (it was calculated from the API and cost CPU cycles to figure it out) and would be very nice to add.

Settle on Database Engine

We are now ready to store some data thanks to @marci4's contributions to the handlers. The big question now is the following:

SQL or NoSQL?

If SQL, what kind? MariaDB / MySQL or Postgress?

If NoSQL, Mongo or Key Value? Etc

Create Connection to MongoDB

Create a database proxy driver which client classes interface with. This allows us to swap database implementations should we need to, and provides us a singular interface to interact with.

Decide on http request package

We need to decide the websocket's http request package for requests to Census / PS2Alerts API etc.

Yes, I know. I would like to directly pull them from the API, but first we need to decide on a http request package

Originally posted by @marci4 in #23

Application / Illegal exceptions are coming through as Census Stream Warnings

E.g. Could not parse Class ID to faction: 0 is being reported as coming from Census when in actual fact it's coming from PS2Alerts.

Winston: When using log filter, application crashes due to not being able to find log levels

@microwavekonijn has a lead on what this is, requires remedials and testing

Refactor to utilize kill_type upon inclusion of Census v0.3.1

Forgot to mention this in the other pr. ps2census v0.3.1 has now a computed property called kill_type which returns a Kill enum with the values Normal, Teamkill, Suicide, and Undetermined(freaking NSO).

Originally posted by @microwavekonijn in #66 (comment)

Update population tracking when PS2Census has been updated with player faction resolution

This will require completion of Census Package faction querying. @microwavekonijn

Originally posted by @Maelstromeous in #105

Get logs to post to Discord

Need error logging to post to Discord so we can monitor for errors. Will likely be very similar code for #37.

Create webhook push to Discord Alert Notification channels

I've created the following channels in the PS2Alerts Discord for notifications of new alerts, which will be hopefully used by the wider community once we start pushing the application back to the public:

#cobalt
#connery
#emerald
#miller
#soltech

PS4 ones to be created when ready.

Integrate Planetside 2 Census Websocket client library

https://www.npmjs.com/package/ps2census

This will act as our connector class, which handles the oddities of census and maintains connections, so we don't have to.

Figure out a way to get the `saveDocument` function in the model factory to enforce model schema interfaces

Currently, the this.factory.saveDocument in for example InstanceHandler.ts does not enforce the schema interface. However, updateOne does as it's using the model directly.

We either have to change saveDocument to use the native model call, or scrap the function altogether and just use insert.

Create ability to query census for additional information the middleman API provided

The nice thing about the middleman API was that it gave us various bits of info, like the outfit name rather than just ID, facility names, etc. It also offloaded caching as well.

We will need to replicate this system, whether it be a separate application to solely handle these requests (the website may need to make use of it as well) or built into the websocket as a module.

It needs to be doing the following:

Accepting a service ID and producing an interface to query census for information
Caching the results, with a TTL of 24 hours. This could be done in Redis, but considering the volume of data we may be caching may have to do it in the database. Bear in mind, our infrastructure is memory limited.

Use TypeScript standards and good practices

Exclude compiled JavaScript from git
Use strict compiler flags
Use typescript-eslint for linting

Exclude compiled JavaScript from git

Emitted JavaScript files are not source files and as such should not be versioned.

Steps to take: Put the dist folder into .gitignore. If the npm start script is needed, ensure it also contains the compilation step.

Use strict compiler flags

Currently we are not using strictNullChecks, which means null and undefined types are silently ignored during type checking. That allows these kinds of errors to happen:

const getNull = (): string | null => {
    return null;
}

console.log(getNull().concat('hello')) // runtime error but no compiler error

There are other compiler flags that aren't used but should be.

Steps to take: All strict compiler checks can be turned on using "strict": true.
On a related note: tsconfig.json could use some refactoring.

Use typescript-eslint for linting

A linter helps catch dangerous, misleading or hard to maintain coding patterns, which are super easy to write in JavaScript/TypeScript. If used to enfore a certain code style it also makes the code easier to read for everyone.

Steps to take: Integrate typescript-eslint into the coding environment, use the recommended rules to avoid mistakes and, if at all possible, agree on some level of code style for the sake of readability and maintainability. When the codebase is cleaned up, apply a pre-push hook that doesn't allow unlinted code to be pushed.

Update ps2census to latest version (0.2.4)

The following things have to be fixed, when we update to the latest ps2census version (current version is 0.1.2):

CensusProxy.ts --> enable 'ContinentUnlock'
ContinentUnlockEvent.ts --> Remove event as unknown cast
Swap from GenericEvent to PS2Event

Refactor Mongoose Initialization to run init() on the models then boot

There is a minor concern with Mongoose in the way it implements indexes. Right now, we're using autoIndex: true in the database options to create unique indexes etc.

However, the recommended method provided by MongoDB is to create indexes first on a on-demand basis. Mongoose recommends the same.

Therefore, we should look at refactoring the database service to pull in all of the models (via bootstrapped models) and run init() on them. Once all of their model.on('index') events have emitted, we consider the database fully booted and continue with the booting of the application.

https://docs.mongodb.com/manual/core/index-creation/#index-build-impact-on-database-performance
https://mongoosejs.com/docs/guide.html#autoIndex