spaceface16518 / strat-collector

Combs through a specific subreddit for posts with over a certain threshold in comments. It then adds them to a database.

Halo Strat Collector Reddit Bot

A simple CLI app that gets posts from our subreddit. It transfers them to a database for our website, linked here.

We use Snoostorm and Snoowrap with Node as a reddit bot. Visit releases to get the most recent copy of the software. Keep in mind, anything with a version tag less than v1.0.0 is not ready for usage. This repository is customized to our settings, but feel free to modify it to fit you needs.

Versioning

vMajor.Minor.patch

Follows the same rules as npm aka semver versioning

For pre-release versioning, refer to issue #15.

Development

Some rules:

1. Start your own branch, master branch deploys to heroku
2. Please make issues and PR's descriptive and insightful. Everyone should know what the change is going to do before they approve it.

Some links may not work yet. You may be directed here if you click on a link that does not work.

strat-collector's Issues

Cannot overwrite 'Strats' model once compiled

Mongo core is throwing this error:

OverWriteModelError: Cannot overwrite 'Strat' model once compiled

Strat is the model I am using for mongoose.

Here is the error text:

OverwriteModelError: Cannot overwrite `Strat` model once compiled.
    at new OverwriteModelError (/Users/AMRIT/GitHub/strat-collector/node_modules/mongoose/lib/error/overwriteModel.js:18:11)
    at Mongoose.model (/Users/AMRIT/GitHub/strat-collector/node_modules/mongoose/lib/index.js:349:13)
    at new module.exports (/Users/AMRIT/GitHub/strat-collector/modules/mongo/models/post_model.js:17:12)
    at module.exports (/Users/AMRIT/GitHub/strat-collector/modules/mongo/find-all.js:17:22)
    at Object.<anonymous> (/Users/AMRIT/GitHub/strat-collector/app.js:39:15)
    at Module._compile (module.js:660:30)
    at Object.Module._extensions..js (module.js:671:10)
    at Module.load (module.js:573:32)
    at tryModuleLoad (module.js:513:12)
    at Function.Module._load (module.js:505:3)
    at Function.Module.runMain (module.js:701:10)
    at startup (bootstrap_node.js:193:16)
    at bootstrap_node.js:617:3

This bug exits as of commit d3475fc

Pre-release versioning

The versioning of pre-releases is a little bit different than full releases. While regular releases will follow semver, pre-releases need to follow a different system. Here is my suggestion:

v0.milestone.version-test_phase.verson

This is how I suggest formatting the pre-release tags.

v

This is the basic version marker, indicating the version. It is ususally included in all releases (pre and full) but npm does not have v marker, so this may be omitted based on GitHubs versioning rules and personal preferance.

0

The 0 indicates that this is a pre-release. The first release will be v1 because it is the first full release. All pre-releases have a 0 to start with.

milestone

This is a numerical indicator of the current milestone. For example, milestone 1 just spits out data, but mileston 2 puts it into a database. Mileston 3 may integrate with a web page or desktop app. Etc.

Version

This indicates the version of the milestone. It is just a minor versioning system of milestone.

test phase

If you have ever seen -alpha or -beta.3 attached to a project version, you know what this is. Testing phases are labeled with greek letters, and have minor versioning attached to it.

Summary

Since the pre-releases are not formatted the same as full releases, there is need for standardization.

Notes

• Probably needs to be added to the README

Validate submission

The bot needs to know which posts are submissions and which are not.

This will help the bot sort through the posts more efficiently. Here are some thoughts I had on this.

Code

The code will be simpler, maybe just an includes statement. The module could double check the validation using an consecutive array method. Maybe?

Some suggestions

This would check to see if the title had the keyword submission in it:

data.title.includes('[submission]')

---

This would remove the [submission] text from the title. This could be invoked right before adding it to data.

if(str.includes('[submission]'){
  str.replace('[submission]', '')
 }

Here is a link for the MDN docs for the replace method.

---

Another method for loping off the submisson text:

if(str.includes('[submission]'){
  str.substring('[submission]'.length)
 }

In this example, we use the length of the [submission] keyword to take it off the beginning of the title.

This method has some advantages:

• Simpler
• Faster
• More reliable

But also has some disadvantages:

• Hard to modify boilerplate
• Assumes the user put the tag on the beginning
  • This actually makes this method less reliable.

It will definitely be easier to code. Here is the docs reference for substring.

---

Format is pretty free, but I would recommend formatting it like I (@Spaceface16518) formatted the export module.

I have formatted it like the following:

module.exports.mainFunction(input: any) {
  if(conditionalFunction(input.part) === "condition") {
    return "return value"
   } else {
    return null
   }
  
  function conditionalFunction(input: any){
    if(input === condition) {
      return true;
   } else {
      return false;
     }
 }

Summary

Essentially, the point of this (hypothetical) module is to make sure that the bot only pulls posts that contain actual strat submissions. There are limited ways to do this, but the real problem is after the check happens. Formatting is also an issue, and not just for consistency.

MongoDB

Intro

MongoDB is a NoSQL databasing service offered by Mongo. Here is a tutorials point tutorial for mongodb. We are going to be using mongodb with node, so here is a tutorial from w3.

Details

MongoDB, being a nosql database, stores key-value pair. This is exactly what we want for this. Their free plan offers 512 MB of storage, which is more than enough for us. Although mongo is written in C++, the databasing format looks almost exactly like JavaScript object literal. MongoDB has many advantages for this project. It could be easily integrated, and it could be easily accessed by more than one app.

With MongoDB, we can push items to a database faster. Because of how the key-value system is formatted, it would not be that much of a stretch to just push the data raw from reddit into the database. This may or may not be a good idea:

Pros	Cons
easy to implement	may not be secure
A fast database service	Limited storage and RAM
written like javascript	Small differences may become confusing
integrates with NodeJS	May become less efficient when integrated

MongoDB is a great way to store strats, and is a great alternative to sql, or other database options. It will take some effort, but mongo can be integrated very nicely.

Summary

MongoDB comes with many advantages, and integrating it into this project is a good idea.

Notes

• MongoDB uses a GUI setup program, so someone has to own it or there has to be a communal password.
• MongoDB integration won't be implemented until v0.2, after the project has been pre-released.

Bot needs own account

Right now, the bot is using the account of @Spaceface16518, which is a security issue. Please make a separate bot account.

Node App.js command not working

While executing routine local tests, I encountered a problem in launch.js. For whatever reason, local tests seem to be breaking at this point. The line says

node app.js

It's nothing special, and it is not something simple like "I'm not building my code before I launch it".

Mongoose keeps querying admin

MongoDB has sets of documents called collections. The app is supposed to query the Test collection to retreve the test posts. Instead it is querying admin.

There are several problems with this. First off, querying the admin collection is not allowed, it is purely for system admin activity and maintainence, not storing data. Second, that is not the right database. Duh.

I've tried a couple of soultions:

  let schema = new mongo.Schema(
    {
      title: String,
      author: { name: String },
      body: String,
      upvotes: Number,
      downvotes: Number,
      gilded: Boolean,
      netUps: Number
    },
    { collection: "Test" }

As well as

  var Test = mongo.model("Test", schema, "Test");

But none of them seem to be working.

Is the typescript branch necessary?

Since the typescript branch literally has no difference with the master branch, and because the current project is modularizing the project, this branch is necessary. Just wanted to ask, @vorana

Reorganization necessary

I could do this own my own, but I can merge it, as a reorg would cause many merge conflicts.

Find all function won't work

The function at the end of app.ts won't work.

Details

This is probably because findAll is a function, and functions need function syntax. While testing, errors complaining of missing credentials. This probably means that no password was entered.

Code

This is how I would recommend correcting the code:

console.log(mongo.findall(cred.mongoPass))

Just pull the mongo password from the credential grabber.

Summary

The findAll function is missing its parameter mongoPass.

Data processing necessary?

Do we really need to include the data processor module? We could always just send the data straight from reddit to mongodb.

Migrate to gulpJS

Choose what data to extract

The streamPost gets tons of data about posts, but what data gets extrapolated needs to be decided. This depends on the future vision for this project, which is in @vorana's wheelhouse.

Environmental variables don't configure

I have dotenv setting environmental variables, but for some reason they are not being processed into the cred object.