foxcomm / highlander Goto Github PK

This is needed to have a correct diff against actual master HEAD

All delete buttons should be the same component and same style.

Currently, integrations/shipstation sub-project is not integrated into the pipeline - it requires additional GOPATH-related changes (similar to middlewarehouse) to be properly built and tested.

cc @jmataya

• Create widget to appear like the mocks.
• Make sure widget integrates with ES queries

Need some help with Shipstation integration for TPG.

I pushed some commits to make it build + containerize:
fc61cd8
176df37

Also here's prepared a Marathon config based on the one from TD environment:
https://github.com/FoxComm/highlander/blob/master/prov-shit/ansible/roles/dev/shipstation/tasks/main.yml

Here's what I get in stderr while running:

I1102 10:25:49.795578  1882 exec.cpp:161] Version: 1.1.0
I1102 10:25:49.809350  1887 exec.cpp:236] Executor registered on agent 253a6cd1-2c3a-4595-b697-f72d7afa91f7-S0
I1102 10:25:49.813031  1886 docker.cpp:809] Running docker -H unix:///var/run/docker.sock run --cpu-shares 256 --memory 67108864 -e MARATHON_APP_VERSION=2016-11-02T10:25:31.950Z -e HOST=10.240.0.55 -e MARATHON_APP_RESOURCE_CPUS=0.25 -e SCHEMA_REGISTRY=http://schema-registry.service.consul:8081 -e MARATHON_APP_DOCKER_IMAGE=docker-stage.foxcommerce.com:5000/shipstation-consumer:master -e API_SECRET=cc8881868232434fb529e4c4f23a9b48 -e PORT_10000=31751 -e MESOS_TASK_ID=shipstation.b9438166-a0e6-11e6-9f0e-02423cbe134c -e PORT=31751 -e MARATHON_APP_RESOURCE_MEM=64.0 -e PORTS=31751 -e ZOOKEEPER=zookeeper.service.consul:2181 -e MARATHON_APP_RESOURCE_DISK=0.0 -e MARATHON_APP_LABELS= -e API_KEY=9b46d64d77664379a35cda3592752bd9 -e MARATHON_APP_ID=/shipstation -e PORT0=31751 -e MESOS_SANDBOX=/var/lib/sandbox -e MESOS_CONTAINER_NAME=mesos-253a6cd1-2c3a-4595-b697-f72d7afa91f7-S0.ba93ce61-df3c-4a05-9c31-5b12be21fb32 -v /var/lib/mesos/slaves/253a6cd1-2c3a-4595-b697-f72d7afa91f7-S0/frameworks/84b7475d-6336-4606-91ee-e17fd3b36418-0000/executors/shipstation.b9438166-a0e6-11e6-9f0e-02423cbe134c/runs/ba93ce61-df3c-4a05-9c31-5b12be21fb32:/var/lib/sandbox --net host --name mesos-253a6cd1-2c3a-4595-b697-f72d7afa91f7-S0.ba93ce61-df3c-4a05-9c31-5b12be21fb32 docker-stage.foxcommerce.com:5000/shipstation-consumer:master
WARNING: Your kernel does not support swap limit capabilities, memory limited without swap.
2016/11/02 10:25:50 Connected to 10.240.0.25:2181
2016/11/02 10:25:50 Authenticated: id=96866210041102358, timeout=6000
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Received a new message from orders_search_view
2016/11/02 10:26:20 Handling order with reference number BR10004
2016/11/02 10:26:20 Unable to create ShipStation order with error Order BR10004 does not have a billing address
panic: Unable to create ShipStation order with error Order BR10004 does not have a billing address

goroutine 55 [running]:
panic(0x85a300, 0xc820f740e0)
	/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
log.Panicf(0xab5ee0, 0x30, 0xc820047c60, 0x1, 0x1)
	/usr/lib/go-1.6/src/log/log.go:327 +0xd8
github.com/FoxComm/highlander/integrations/shipstation/consumers.OrderConsumer.Handler(0xa69070, 0x12, 0xc82010ee60, 0x7f8858c4c448, 0xc82007d2d8, 0x0, 0x0)
	/home/ejik/go/src/github.com/FoxComm/highlander/integrations/shipstation/consumers/order_consumer.go:43 +0x4f0
github.com/FoxComm/highlander/integrations/shipstation/consumers.(OrderConsumer).Handler-fm(0x7f8858c4c448, 0xc82007d2d8, 0x0, 0x0)
	/home/ejik/go/src/github.com/FoxComm/highlander/integrations/shipstation/main.go:30 +0x57
github.com/FoxComm/highlander/integrations/shipstation/vendor/github.com/FoxComm/metamorphosis.createStrategy.func1(0xc82052a7e0, 0xc82009da80, 0xc8204f4780, 0x12, 0x0, 0x19, 0x0, 0x0)
	/home/ejik/go/src/github.com/FoxComm/highlander/integrations/shipstation/vendor/github.com/FoxComm/metamorphosis/consumer.go:82 +0x1a7
github.com/FoxComm/highlander/integrations/shipstation/vendor/github.com/elodina/go_kafka_client.(*Worker).Start.func1(0xc82052a7e0, 0xc82000b494)
	/home/ejik/go/src/github.com/FoxComm/highlander/integrations/shipstation/vendor/github.com/elodina/go_kafka_client/workers.go:404 +0x150
created by github.com/FoxComm/highlander/integrations/shipstation/vendor/github.com/elodina/go_kafka_client.(*Worker).Start
	/home/ejik/go/src/github.com/FoxComm/highlander/integrations/shipstation/vendor/github.com/elodina/go_kafka_client/workers.go:417 +0x57

When we see images in the admin ui [eg. product listing], we are currently not optimizing image size, so bandwidth and page performance can be a problem. Imgix is a perfect tool to do this, we should always render admin images optimized for resize and device.

Project name: Goldrush

Goal: Make building the local environment very easy when you are actively developing one or more of the Fox modules.

This tests are now commented out, because they're causing troubles with stage provisioning

#62
dc60f65

A new endpoint handler that hits our cross-sell service to get a product to suggest [based on last product purchased for now, and filtering out all products already purchased], and triggers an event that an upsell has been suggested to the user.

Service must also make sure we have all the requirements for 1-click checkout [phone#, validCC, shipping & billing addresses] communicating with dexter. If we can't recommend a product, we must send an appropriate response to the client, so admin ui can message the admin who clicked the button.

API endpoint to receive "suggest" action

• Receive POST from admin via ajax
• ask Xsell service for a product to suggest
• design "suggest product" name and schema
• Call dexter

Request arguments:

customer_id

Response arguments:

success | error

Information to dexter must contain at minimum:

- 1 product with minimal model:
  - ID
  - name
  - short description
  - 1 imageurl, etc
- user phone#
- probably also user ID for completeness

No we have Cache-Control: max-age=0 for all static what we have in every site what we have now (Ashes and all storefronts) and server doesn't handle 'if-modified-since' header.

What does this mean ?

It means browser should load all static files each time customer visit page. This in turn increases page load time which is not good for sales and UX.

In order to get rid of this nginx should serve static files by themselves which means nginx should have access to projects static files.

Another way is serving static files through CDN, which requires some changes in sites build process.

Personally, I'd like more first way because it's more isolated and independent. And after we do the first, we can make a second.

Links about caching headers:

• https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching
• https://www.mnot.net/cache_docs/

in russian:

• http://html5.by/blog/cache/

Advanced guides for the developer portal.

Basic API reference material for the Developer Portal.

Hey–hey!

So yesterday, before going to sleep, and after work and chatting with @mempko, I did my homework and Google™’ed a bit about Design by Contract in the context of Functional Programming and software
correctness etc. (I also wrote this piece yesterday, just publishing it today. I hoped for some sleep enlightenment badly.)

And, frankly, I’m a bit lost. :( I tried, believe me, I really did—that was almost the only thing I did after logging off work—and I questioned everything I learned so far. (Imagining all your current assumptions on some matter to be false is a great thought experiment/exercise! ❤️ ).

But I can’t see how throwing assertion exceptions in the middle of an execution of a production program can be beneficial—when compared to encoding possible errors in appropriate return values if your language’s type system can encode them. Isn’t this, the Referential Transparency—which, nb., exceptions violate—the foundation of Functional Programming?

I even looked through Cats and Scalaz code for reference—do we agree that these guys in general know how to encode correct Scala programs? Results:

• In Cats there’s 1 require, 28 asserts, 0 ensurings, 0 assumings — and all of them in the tests, not in runtime. And you can freely throw in the tests, that’s how they’re implemented in Scala anyway. They also only have 2 throws in core, for NonEmptyList construction from Nil, and similar for NonEmptyVector. All this in 16.5 kLOC.

• In Scalaz, they have 1 require in the build definition, so not runtime. Also, 208 asserts in throw-away example code (i.e. how-to-use-scalaz-and-what-will-be-true-when-you-do subproject) and 4 in tests, 3 throws, 0 assuming, and 0 ensuring in 40.5 kLOC.

If anything, in the FP community that I’m following (again, I might be looking up to wrong people), exceptions are highly discouraged…

I can only hope you’ll excuse this somewhat dramatic tone, but I’m really torn. What to do, what to think, and most importantly, how to program our services?

More egoistically speaking, I really liked the feeling of knowing what I’m doing, but… it’s gone since the yesterday’s chat. :( Can I have it back?… 😿 Anyone?…

/cc @jmataya @mempko @Anna-ZZZ @kpashka @narma @vkorbut

Admin finds a customer record and clicks "Suggest Upsell", which POSTs to a new API endpoint. [Ultimately this event could be triggered by a purchase for post-purchase upsell, or a regular interval for a subscription business, abandoned cart timeout, etc etc. But for now a button in the admin will allow us to control the flow of the demo]

• Button in admin on customer record to send "suggest upsell" to customer, new API endpoint as per #1078

FoxCommerce needs a robust developer portal that is stays up-to-date as part of our daily workflow.

Lots to do in order to get this done for 1.0.

• Get basic portal operational
• Add tutorial content
• Put in basic authentication

Future Roadmap:

• Add roadmap section
• Put behind Fox Authentication
• Import APIB blueprint
• Make part of the build process

https://docs.google.com/document/d/18GMxZ1_VxbeXF6EOr0xEdz0_Ye6fXP5qPyjhRnqP4cI/edit#heading=h.i3wt1qemqb5d

https://docs.google.com/document/d/1r99FOGnlSGu5r0sop4X-7UYSMYEFjiGKQqe6duk6rdA/edit

https://app.zeplin.io/project.html#pid=588e727b9fac123a15dc80c3

Admin users should be able to export data from all lists in csv format.

• Will stream from the browser in native download format [ie, no ui around the download itself].
• For MVP, if more than 10,000 records [what ES can handle], we will generate a user message saying there are too many records to export.
• Once we start hitting this limit on client sites, we can implement a background task and email results to user.

User Stories

1. Selects a few items, clicks bulk actions, exports just those selected items
2. Clicks download icon button at right, exports the entire search results [all pages]

Priority of Interfaces

• dynamic customer groups
• coupon codes
• products

Docs

• Mocks on Zeplin

To Dos

• Make "Toggle Columns" button smaller - just icon; no text
• Add the "export entire list" button
• Add option to "export selected" to the bulk actions menu

We want to allow store owners to collect payment via ApplePay. Hopefully this should be straightforward, here are some relevant links:

• Apple Pay marketing site
• Apple Pay Developer home
• Developer getting started

Subscribes to the "upsell has been suggested" event, takes the args and sends them to Dexter webhook endpoint to start the customer conversation.

Dev Dexter endpoint:

http://bots.daniel.robopipes.com/api/v2/bot/e30c5824-0c18-40ab-93d7-0f85c8822b61/command/upsell/9178646471/?api_key=e25b39ca4455428691f521c5afe3421b

• TODO(@stresslimit): get from Dexter the schema to add user & product metadata like product name & id. I hope it's as simple as arbitrary GET vars.

Phase 1: Living Styleguide

Goal: Set up the basic framework for how we are going to show component examples

• Set up in its repo, install a storybooking solution
• Go component by component, remove css modules and install simple, understandable CSS

We need good documentation and a clear way to develop and maintain components
in isolation, so we implement a demo of all components. Simplistic example from my time at
littleBits can be found here.

Phase 2: The Basics

Goal: MVP importing to a minimal storefront repo, allowing to spin up multiple storefronts easily.
Non-goals: not immediately tackling the more complex items like a robust inheritance mechanism

We can just do dumb overrides by changing the import statements from npm to local path, basically whatever is required to get it up and running for now.

Phase 3:

Goal: Allow storefront developers to extend the core storefront.js elements as they build out more customized experiences

We should incrementally work ways that store owners can customize their stores based on storefront.js framework.

1. Extend components: Ideally we can override just display components, and keep using the same core smart/data components. Suggested way to roll these out could be:

• PDP qty selector
• add to cart button [loading state]

2. Extend routes:

• simple routes [/my-single-handler-route, /pages/my-page]
• dynamic root routes [/my-page, /my-blog-post]

Appendix

Based on this doc

Initial Research and Ideation (from Pavel)

The main difference between us and Heroku is that we have a specific stack with microservices connected to each other in very specific way, while Heroku is just a generic wrapper around AWS. That's our power, yet that's also our weakness.

We need to improve our deployment speed to make it as sexy as Heroku, current 30 minutes appliance provisioning is unacceptable. So basically, improving developer experience overall will lead us to more streamlined automation.

A couple of ideas to improve that:

• #1245 Automated production deployments for TD/TPG staging and production environments.
• #1243 Super Fat Base Images, containing everything. A lot of time is being spent by Ansible to handle SSH connection over the VPN across the ocean for very simplified tasks like "upload a similar Marathon JSON files multiple times". If all necessary files, databases and other stuff would be already there, it would save a LOT of time.
• #1242 Marathon Application Groups. A group of containers with shared dependencies that are deployed together. There is no need to wait for each application healthcheck anymore, we can just wait for green group status.
• #1881 Using Nixy instead of Consul Template (and Mesos Consul). Unfortunately, only commercial version of Nginx can load upstreams dynamically from Consul, we're using Consul Template as a workaround for that, but there is no way to trigger it's reload from Nginx itself, which leads to minor downtime and 502 errors. Nixy is doing the same thing as Consul Template, except, it is continuously reading Marathon applications via it's Event Bus, without man-in-the-middle apps like marathon-consul.
• Terraform 0.9.x State Environments. We can automate the launch of similar instances just by creating a new environment that will catch-up necessary configuration. Additionally, it would be cool to have state backend with state locking (e.g. in a separate Consul cluster) to avoid race conditions.
• CLI App. At very early days, Heroku didn't have so nice web UI, their main wow-factor generator among Rails community was a tool called Heroku Toolbelt, which is just sexier version of AWS CLI and Google Cloud SDK. Potential responsibilities of Fox CLI tool could be - integrating with GitHub webhooks by providing API Token, calling webhook server manually from command-line etc.
• #1138 Stability. We need to handle software recovery after instance shutdown & restart. AFAIK, main problems were related to Confluent Platform and Mesosphere. (blocked by MESOS-6223 and MARATHON-2694)

The synergy of various components above could lead us to interesting things - e.g. if we move all our Consul services to Docker (could be challenging because of things like Bottledwater) and launch everything in a single pod with internal service discovery (via dockerized Consul, for example) - it could become a true one-click deployment even with kind-of version of multi-tenancy.

AFAIK, Max's team is already using dockerized version of Neo4j with persistent volumes.

Create an endpoint in anthill that will accept a customer ID and return a list of products to suggest. The list of products should contain whatever data is stored in ES products_catalog_view.

The recommended products should not contain previously purchased products.

This new endpoint will be called by the suggester service so that it can recommend products via messaging (dexter)

We need a review and refactor of admin ui components, starting with the basics of a living styleguide where we can incrementally consolidate redundant and inconsistent ui code.

@jmataya help flesh this out

Internet-accessible endpoint protected by some simple secret token, which receives user phone# [perhaps user ID], product ID, and POSTs to Fox API to create a cart, add the product in question and auto-checkout with default settings [card & address].

Notes from Dexter:
https://gist.github.com/ilkovich/0fc0b634c328b77a13ad25d52985f700

NOTE: This is mostly throw-away code for prototype only.

Ultimately we should make this part of our BYOC infra, but for now it's probably quicker [and fun] to make it a google cloud function.

GET
Request args:

userPhone, productID,securityToken [hard-coded at dexter and here, or google oauth token hard-coded at dexter]

Response args:

success | error

We need an awesome one.

What we want:

• Beautiful documentation that is centralized, easy to consume, and used by our team on a daily basis.
• A redacted/filtered version that is automatically built/published for our customers consumption as well.

Eggcrate should compute averages for a product correctly.

Example correct psudo code.

sales = henhouse('products.sales', time_period, frequency)

total_active = 0
for sale in sales
    total_active += henhouse('products.active', sale.period)

average_sales_per_product = sales.total / total_active

We can either do the loop above in eggcrate or we can add a new parameter to henhouse called mask_key

/diff?keys=active&a=<start>&b=<end>&step=<frequency>&mask_key=<some key>

The response will provide total_active from the loop above

Demo customer interaction to purchase via SMS text message. User flow:

1. Suggest Upsell admin button
   Admin finds a customer record and clicks "Suggest Upsell", which POSTs to a new API endpoint. [Ultimately this event could be triggered by a purchase for post-purchase upsell, or a regular interval for a subscription business, abandoned cart timeout, etc etc. But for now a button in the admin will allow us to control the flow of the demo]
2. Suggest Upsell endpoint
   The new endpoint handler then hits our cross-sell service to get a product to suggest [based on last product purchased for now, and filtering out all products already purchased], and triggers an event that an upsell has been suggested to the user.
3. Upsell Suggested event listener
   Subscribes to the "upsell has been suggested" event, takes the args and sends them to Dexter webhook endpoint to start the customer conversation.
4. Customer SMS reply handler
   Internet-accessible endpoint protected by some simple secret token, which receives user phone# [perhaps user ID], product ID, and POSTs to Fox API to create a cart, add the product in question and auto-checkout with default settings [card & address]. Probably a google cloud function.

Suggest Upsell admin button

• Button in admin on customer record to send "suggest upsell" to customer

Suggest Upsell endpoint

API endpoint to receive "suggest" action

• Receive POST from admin via ajax
• ask Xsell service for a product to suggest
• design "suggest product" name and schema
• create "suggest product" event

Event must contain at minimum:

- 1 product with minimal model:
  - ID
  - name
  - short description
  - 1 imageurl, etc
- user phone#
- probably also user ID for completeness

Upsell Suggested event listener

• Listens to "suggest upsell" event and POSTs to Dexter

Customer SMS reply handler

• Receives "yes" from Dexter webhook and creates the new upsell order via core Fox API

First step here is to investigate ways we can do this, a likely candidate is React Storybook

In order for eggcrate to return valid product statistics, it needs to know the total number of active products within a time range.

Currently it gets the count of active products from henhouse however this value is manually set.

https://github.com/FoxComm/highlander/blob/master/intelligence/eggcrate/src/services/productStats.go#L49-L55

We either need to create a consumer which updates the active count in henhouse or change eggcrate to query ElasticSearch.