I am being charged for used space every day yet i have no buckets in my account, why is that?

See spec here

Replace the .jshintrc with the Storj .eslintrc from core repo and add it to the NPM scripts.

Should file names be included in billing?
Once a contract expires, shards are deleted but file names don't, and remain visible inside the bucket.

An attacker could upload files with really long names and small size, so that when contracts expire he is still able to read data for free.

1. Currently farmer payouts are higher than the amount renters pay, which leads to abuse by users actively downloading from themselves.
2. Farmers are still abusing the base payout by running allot of nodes.

I'm get credit $10 to my Bridge account by clicking on the referral link.

And have a weird "usage" on the Billing page:

However my usage history never close to it:

If the process for creating debits, credits and referral credits is run twice, duplicate entries should not be created. Existing entries should be updated with the data in the current run and new entries should be added.

This should all be done in such a way that there will never be a case that an entry has changed or removed such that it cannot be found and is added causing a miscalculation in balance.

Over the last few months users that have a CC added to app.storj.io and are uploading beyond the free tier are not being charged, thus essentially uploading for free. This includes users with a virtual and expired CC on app.storj.io.

Related to #296 on Storj/bridge - API lacks protection from abuse - We need to beef up the security of the billing server

@dylanlott , @pgerbes1 , @bryanchriswhite .

Normal conditions:

• I upload 200GB/day
• "GB of storage used" increases every day as i constantly upload more data and don't delete any.
• "Current usage" increases every day as i have to pay daily for my uploaded data.
• "Storage" in "Usage This Month" increases every day.
• "Bandwidth stays close to zero as i don't download anything, and what i download doesn't complete successfully due to the known bridge stability issues.

Observations:

• Current usage stopped increasing.
• "Bandwidth" in "Usage this Month" increased exponentially without any actual downloading taking place from my account.
• "Description" in "Billing History" dropped to "<0.01 GB of storage used", while i (1) Did not delete any data and (2) I am still uploading every single day.

screenshots

Currently S3 - infrequent access is cheaper on amazon than the Storj price, which can lead to potential customers choosing S3 over Storj. There are many farmers that want to store allot of data, but don't like Storj using up allot of bandwidth. For those farmers storing infrequent access data would be a solution.

S3 infrequent access costs: $0.0125 / GB

User should be able to refer another user by email and get their referral link for copying and pasting elsewhere.

Create e2e and unit testing framework for billing.

• E2E tests with PioneerJS
• Unit tests with Mocha/Chai (same format as Bridge)

Currently it is not attractive to use Storj mainly because it offers nothing that other well known reputable storage companies already provide, Storj does not have a established reputation, its expensive for consumers and even industry, transfers are slow, the network is unreliable and there is a single point of failure of the Storj service.

As explained in detail in SIP-006 farmer benchmarking and selection based on different performance/reliability metrics (bandwidth, geographical location, uptime, timeout-rate...) is the way forward not only to make Storj more efficient but also to allow Storj Labs to target different audiences/landscapes within the Storage industry.

In order for Storj to survive it will have to specialize from the current uniform and on average low quality storage cloud to attract specific clients that are after a specific performance quality or metric of the Storj network.

This implies creating performance profiles renters can adjust after registering to app.storj.io, and adjacent billing for that specific user selected profile. These profiles are dynamically priced based on prices of other Storage competitors.

An example of the implementation:

• Renters create a storage profile of their choice on app.storj.io, then exports a settings file with the parameters that can be imported into say Libstorj or FileZilla, farmers will be automatically selected based on the parameters in the config file.
• Farmer creates a storage profile, the bridge remembers the profile linked to a specific account and only selects farmers that satisfy this profile, independently where users login, as long as they use the specific account.

We need to create a better test flow and framework for billing for both staging and production pushes.

Giving users that pay with the STORJ token a discount (e.g. 10-20%) will incentivize people to use and or hold the STORJ tokens, this would settle the debate about what advantage of using the STORJ token has over just using fiat. The main complaint users have (mainly token buyers) is that the STORJ token can't really be used for anything within the STORJ ecosystem and thus should have no intrinsic value or use.

Currently i am paying 15$/TB for 6 mirrors per shard, i want 3 mirrors and i want to pay 7.5$/TB of used space.

Credits should expire when past their expiration date.

This should happen only at the end of a billing cycle, right after credits are used.

Thing that should not happen:

• $10 invoice due on 21st
• $5 credit expired on the 15th
• feelsbadman.jpg

This strengtheners the "respect" and desire so to say of Storj labs towards farmers, basically giving them some extra because they are a farmer which is a critical aspect of the network. This also creates another incentive to become a farmer and support the network.

Currently manually paying with STORJ tokens for storage is way too hard.

These rate limits are a huge pain for everyone and don't bring any advantages, only disadvantages, like loosing potential customers.

This results in customers being charged for files they no longer have stored.

Tasks

• Catch all rejected promises from the Promise.all's
• Fix the error thrown by billing for failed Stripe requests

Summary

When generating debits, if there is an error when calling out to the Stripe API, the billing-importer script does not catch it.

(node:506) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: [object Object]

This could be caught at the Promise.all like so...

        cursor.on('end', () => {
          Promise.all(insertPromises).then(() => {
            resolve(true)
          })
          .catch((err) => {
            console.log('Error processing debits', err);
          })
        })

But currently the err object returned is not the proper format and is not usable.

Error: [object Object]
    at Request._callback (/root/lib/utils/billing-client.js:42:49)
    at Request.self.callback (/root/node_modules/request/request.js:188:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/root/node_modules/request/request.js:1171:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (/root/node_modules/request/request.js:1091:12)
    at IncomingMessage.g (events.js:291:16)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Have a minimum billed storage length for the case that files are uploaded and immediately deleted: https://github.com/Storj/dev-meetings/blob/master/2017-8-9-summary.md#users-flooding-network-with-data-and-deleting-it

Objective

In order to test whether the stripe queue changes are working as expected we need to test that given some storj usage (storage events, bucketentries, and bucket(s)), that debits, credits, and stripe invoice items are accurately generated for their respective user.

Testing Environment / Services

Currently everything is deployed to staging that is necessary for the test. These services are:

• Staging Billing: billing.staging.storj.io - receiving api calls from billing-importer and from stripe
• Staging Bridge GUI: app.staging.storj.io (not required, just useful) - can be used to look at a users transactions and period total easily

Relevant files:

(There's currently a decent amount of logging in a few of these files. I haven't yet replaced console.logs with logger.debugs yet in many places. I would recommend continuing to use a ubiquitous logger and replace console.logs as you find them.)

Billing:

• lib/server/routes/credits.js
• lib/server/routes/debits.js

Service-storage-models:

• lib/models/payment-processor-adapters/stripe.js - contains all the stripe specific logic for billing; i.e. where all (there may be some exceptions that are expected to be factored out later) API calls to stripe are made from
• lib/vendor/stripe.js - the most recently changed part of billing, contains the stripe api queue / proxy implementation

Methodology

*indicates that this step can probably be done easiest by using the staging bridge-gui

1. Create a new storj user*.

2. Add stripe payment processor to that user*.
   If you want payments to succeed in stripe test env you want to use this card number (see stripe's docs): 4000000000000077

3. Create multiple days worth of usage data in the staging database.
   Documents required for proper debit generation:

   • bucketentry
   • bucket
   • storageevent
   • frames (maybe, not totally sure)

   Data generation:

   1. Create multiple buckets and bucket entries via the cli and/or bridge-gui (for buckets).
   2. Download each of your entries once via the cli to generate download storage events.
   3. You can now duplicate and manipulate these storage events to generate usage data for multiple days, changing the timestamp.

4. Generate debits for current billing period.

   This can be done easily by kubectl execing (I use an alias kube in my examples) into the running staging billing-importer container and executing the bin/create-debits.js bin with the appropriate args:

   kube and kns are aliases I copied from @phutchins and you can find it here as well as it's bash completion

   export KUBECONFIG=<path to your kubeconfig.yml>
   kns storj-staging
   kube exec -it $(kube get pods|grep billing-importer|cut -d' ' -f1) /bin/bash
    
   #--- now in billing-importer shell ---#
    
   bin/create-debits.js --help

   Usage: create-debits [options]
    
   Options:
    
     -h, --help           output usage information
     -V, --version        output the version number
     -b, --begin [time]   Begin Time in ISO Date format EX: 2017-03-27 - Default: Today
     -d, --days   Number of days to generate debits for starting at Begin Time - Default: 1
     -R, --remove         Remove existing debits within the selected date range before generation (not yet implemented)
     -y, --yes            Automatically say yes to all prompts

5. Verify that the total of the given stripe user's pending invoice items matches the sum of the storj debit amounts for the corresponding user (and billing period). NOTE: Stripe has a miminum of 1 cent for invoice items, so the invoice item and debit amount sums should be the same to a cent.

   Example stripe customer "Pending Invoice Items" section:

   Pending invoice item(s) present

   

   No Pending invoice item(s) present

   

   Honestly not sure

   

6. Generate an invoice to trigger stripe webhooks to test the billing credit route's credit creation:

   

7. Verify that the storj user has a new credit with paid_amount equal to 0, and invoiced_amount equal to the cent-floor of the stripe invoice total.

   You can go to the "Events & Logs" section of the stripe UI and select the entries corresponding to the invoice.created event to get details on whether the webhook has been triggered yet, what the response was, and gives you the ability to re-trigger if you wish.

   After stripe's webhook(s) have triggered, you can use/refresh bridge-gui to verify the existence of a new credit and to verify the paid_amount but bridge-gui doesn't display the invoiced_amount of debits. You'll probably have to look at the database to get this info.

8. "Pay" the invoice via stripe to test the billing credit route's credit confirmation:

   

9. Verify the storj user's new credit was updated with a paid_amount equal to the invoiced_amount (and by extension stripe's invoice total)*.

   You can again look at the "Events & Logs" section and look for the entries corresponding to the invoice.payment_succeeded or invoice.payment_failed events, the webhook request, response, etc.

   Again, after stripe's webhook(s) have triggered, you can use/refresh bridge-gui to verify the change in paid_amount since this is what's rendered into the "Amount" column of the transactions list/table.

Save Credit Card Button doesn't work
https://docs.storj.io/v1.1/discuss/5a4802168cf70200122a6cde
https://docs.storj.io/v1.1/discuss/5a484d61e9ff3d0012ab776f

Looks like with bitcoin payments (maybe also credit card) isFreeTier is not automatically set to false yet and requires manually setting it.

Most cloud storage providers offer lower prices the more the customers stores or the more they download the data. If we take S3 as a example the price of storing over 500TB of data is $0.021 / GB, then $0.050 per GB for transferring data outside of S3 for over 350TB, this means that S3 is the same price as Storj at this point, except for a minor difference in storage costs. These price differences are not good enough for such a large customer to consider Storj as a viable option, the prices have to be lower.

Creating a similar scale as in https://blog.cloudability.com/aws-s3-understanding-cloud-storage-costs-to-save/ should work.

There are currently zero unit tests, include unit test coverage for critical areas of functionality.

When I click on remove card the screen flashes, and the button says "Submitting." After about 5 seconds, the card returns and the button says "Remove Card"

Here is the JSON response from the server:

{  
   "data":{  
      "paymentProcessor":{  
         "id":"XXXXX",
         "name":"STRIPE",
         "billingDate":7,
         "defaultPaymentMethod":{  
            "merchant":"Discover",
            "lastFour":"XXXX",
            "id":"card_1XXXXXX"
         },
         "error":null
      }
   }
}

These are duplicated from bridge, and are no necessary here, as from discussion.

Users are not getting set to paid tier status after buying credit with STORJ.
This affects their ability to upload data after they are erroneously rate limited.