rohannagar / lightning

Real-Time Social Media REST API

Lightning

Lightning is a real-time social media REST API, to be used in conjunction with Thunder. Lightning provides methods to request data from various 3rd party social media and cloud services, including Facebook and Twitter.

• Endpoints
• Running Locally
• Testing
• Changelog

Running Locally

• Requirements
  • Java 1.8
  • Maven 3.3.3

First, fork this repo on GitHub. Then, clone your forked repo onto your machine.

$ git clone YOUR-FORK-URL

Compile and package the source code with Maven.

$ mvn package

Run the packaged jar with the server argument.

$ java -jar application/target/application-*.jar server

Lightning should now be running on localhost port 9000.

Testing

There is a Python testing script available in the scripts directory. To run this script, make sure you are in the base lightning directory and run the following command.

$ python scripts/tester.py

There are multiple optional command line arguments for the testing script. These are described in the table below, along with their default values. Additionally, when running the script from the command line, adding the -h option will display a help message with all optional arguments.

Flag	Description	Default Value
-h --help	Display a help message.	----
-e --endpoint	The endpoint to connect to lightning with.	http://localhost:9000
-m --email	The email to make requests for.	[email protected]
-p --password	The password of the user that matches the username. Should not be a hashed password.	password
-a --auth	The basic authentication credentials in the form {app_name}:{app_secret}	application:secret
-v --verbose	Provide more output information.	False

When adding new endpoints to lightning, be sure to include a new TestCase for that endpoint in the tester.py script.

Another testing option is to use HTTPie. See the wiki for each individual command.

lightning's Issues

Remove 3rd Party Library Usage from Resources

For example, we should not be catching a restfb exception in the FacebookResource class. This should be done in the FacebookService class.

Add JitPack repository for Thunder

Since migrating Thunder to JitPack instead of S3, the build is now errored due to the S3 artifacts being deleted.

Dependabot couldn't authenticate with https://jitpack.io

Dependabot couldn't authenticate with https://jitpack.io.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

Implement Text-Only Publishing for Facebook

Currently we can publish photos or videos, but we also should be able to post only text. The publish method in FacebookResource should have the ability to do any type of publishing based on what it receives.

Add JSON Serialization Tests for API Models

We need unit tests for all of the models in the API.

Improve Thunder Error Handling

Right now error handling Thunder responses is confusing and can result in unknown errors.

Handle error responses from Thunder

When making network calls to Thunder e.g. thunderClient.getUser(username), we should handle the cases where Thunder is down or we are unauthorized. Currently an uncaught exception is thrown.

Create Client Library

A Java client library would add the feature to easily call each endpoint from another Java application in the future. This would be important if we built another application, such as a scheduler.

Dependabot couldn't authenticate with https://jitpack.io

Dependabot couldn't authenticate with https://jitpack.io.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

Dependency Upgrades and Dependabot Integration

Before turning on the Dependabot integration, we should upgrade known dependencies and fix known problems with upgrading certain dependencies (carried over from experience with Thunder).

Dependabot couldn't authenticate with https://jitpack.io

Dependabot couldn't authenticate with https://jitpack.io.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

Cache Platform Service instances per user

Each time a resource method is called, we create a new service instance using our factories. This is fine, but does add some additional overhead.

As a potential improvement, we can cache these instances for re-use.

This is currently low priority because there is not enough traffic that this would cause a heavy slow-down.

Improve Unit Test Clarity and Organization

The unit tests, especially for the Resources, are confusing and not very intuitive. We can make them clearer and more organized.

We can look here for more information on what is available to use in Mockito 2.x: https://github.com/mockito/mockito/wiki/What%27s-new-in-Mockito-2

Write cURL Testing Script

We should have a script that we run to test each endpoint so that we do not have to manually test each endpoint. This will become even more necessary as we add more platforms and endpoints.

Change Facebook /oauthUrl to return JSON

To make it consistent with Twitter's endpoint, we need to return JSON instead of a String.

Lightning Email Validation

Lightning needs to validate an email by sending an email to a newly registered user or newly registered email for an existing user. Once Lightning can validate that a user has access to the registered email the registration process can finish or an email can be added as a valid address for the account.

The process will go something like this:

1. Start the validation process by creating a new user and denoting the associated email as unverified. If the user already exists then just add a new email as unverified.
2. Generate a unique hash and store it along side the unconfirmed field in the DB.
3. Append the hash as a query param to a url that points to a validation endpoint on Lightning and send the url to the address you want to validate.
4. When lightning receives an email with a hash, parse the hash and confirm the user is associated with the given hash.
5. Mark the email as valid in thunder.

Note that a user account can be viewed as unverified if there exists only one email and it's not verified. This can be useful for tracking down accounts that are registered but never verified.

The end goal is to allow account recovery and general contact with our user base.

• Add email validation field associated with each unique address in the DB
• Add email random hash associated with each unique address in the DB
• Ability to generate a random hash to store in the DB
• Ability to create a URL containing an emails associated hash value as a query param
• Ability to parse the URL containing a hash value and verify it belongs to a certain users email
• Ability to mark individual user emails as invalid/valid
• Ability to store an associated hash value along with each unique email on a user account

Improve Logging

There needs to be much more detailed logging everywhere.

Twitter Upload is Limited

Currently Twitter upload is limited to a small filesize. We can fix this by using chunked upload, but Twitter4J does not support it, as development has stopped on it.

Our end goal should be to build our own Twitter Java SDK and use that moving forward. For a short-term workaround, we can fork Twitter4J and pull in some changes that people have made in PRs. Then we can pull in the new jar using Jitpack.

Change return type if authentication credentials to Thunder are incorrect

Currently, Lightning propagates error responses from Thunder. If Lightning receives a 401 Unauthorized from Thunder, it should respond to its caller with a 500 Internal Service Error, since the user calling Lightning is not unauthorized, the server itself is unauthorized.

Change /extendedToken to /authorize

Changing to /authorize would be clearer naming

Upgrade RestFB Version

We badly need to update RestFB so that we can get the correct OAuth URL.

Remove Dropbox-Related Code

Dropbox is no longer going to be used. In order to maintain a clean code base, we should remove it. It can always be added back in if necessary.

Lightning Swift SDK for iOS

Would be nice to have an SDK at some point that handles requests and OAuth stuff