featurebasedb / infrastructure Goto Github PK

View Code? Open in Web Editor NEW

5.0 5.0 8.0 259 KB

Assorted tools for operations and deployment of infrastructure

License: BSD 3-Clause "New" or "Revised" License

Smarty 0.61% Makefile 13.40% Python 16.59% Shell 7.96% HCL 58.74% Go 2.70%

infrastructure's People

Contributors

Stargazers

Watchers

Forkers

jaffee codysoyland tgruben seebs m-schumann shaqque alanbernstein asmtal

infrastructure's Issues

use default linux images on each cloud

Switching from Ubuntu to Oracle Linux in OCI netted some perf gains. Probably one can expect the best performance and hardware support in general from cloud-specific linux images (e.g. Amazon Linux). It's not clear whether Azure has a single linux image which would be considered its default.

Automatically shut down test cluster resources after configurable TTL

@codysoyland commented on Fri Dec 16 2016

@codysoyland commented on Tue Dec 20 2016

This is proving to be more difficult than anticipated. AWS has a blog post that shows a method of doing this: https://aws.amazon.com/blogs/devops/scheduling-automatic-deletion-of-application-environments/

But the AWS method requires use of an EC2 instance with the cfn tools installed (such as Amazon Linux) and I would like to avoid having extra software installed on the pilosa machines. I would like to use Lambda to schedule the deletion, but I haven't found a clear way to schedule a lambda job.

I will Icebox this for now and we can discuss options in the future.

Ensure max file handles (ulimit -n) extended for all cloud providers

Just a reminder to make sure the max file handles is increased to at least 32K, maybe even 64k from the default 1k on ubuntu boxen for all pilosa node instances

Error building bandwidth

Following the guide in the terraform examples directory, I encountered an error running make provision at TASK [build bandwidth]. Output looked like fatal: [pilosa0]: FAILED! => and then a compact JSON error message, expanded below. This doesn't seem to be blocking (if you also run make provision-agent) as it happens near the end of make provision, reporting here to look into later.

{
  "changed": false,
  "cmd": "/usr/bin/make bandwidth64",
  "msg": "ar: creating OOC64.a\nroutines-x86-64bit.asm:723: error: parser: instruction expected\nroutines-x86-64bit.asm:725: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:725: error: parser: instruction expected\nroutines-x86-64bit.asm:727: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:727: error: parser: instruction expected\nroutines-x86-64bit.asm:729: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:729: error: parser: instruction expected\nroutines-x86-64bit.asm:739: error: parser: instruction expected\nroutines-x86-64bit.asm:740: error: symbol `vmovdqa64' redefined\nroutines-x86-64bit.asm:740: error: parser: instruction expected\nroutines-x86-64bit.asm:741: error: symbol `vmovdqa64' redefined\nroutines-x86-64bit.asm:741: error: parser: instruction expected\nroutines-x86-64bit.asm:742: error: symbol `vmovdqa64' redefined\nroutines-x86-64bit.asm:742: error: parser: instruction expected\nroutines-x86-64bit.asm:753: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:753: error: parser: instruction expected\nroutines-x86-64bit.asm:755: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:755: error: parser: instruction expected\nroutines-x86-64bit.asm:757: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:757: error: parser: instruction expected\nroutines-x86-64bit.asm:759: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:759: error: parser: instruction expected\nmake: *** [bandwidth64] Error 1",
  "rc": 2,
  "stderr": "ar: creating OOC64.a\nroutines-x86-64bit.asm:723: error: parser: instruction expected\nroutines-x86-64bit.asm:725: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:725: error: parser: instruction expected\nroutines-x86-64bit.asm:727: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:727: error: parser: instruction expected\nroutines-x86-64bit.asm:729: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:729: error: parser: instruction expected\nroutines-x86-64bit.asm:739: error: parser: instruction expected\nroutines-x86-64bit.asm:740: error: symbol `vmovdqa64' redefined\nroutines-x86-64bit.asm:740: error: parser: instruction expected\nroutines-x86-64bit.asm:741: error: symbol `vmovdqa64' redefined\nroutines-x86-64bit.asm:741: error: parser: instruction expected\nroutines-x86-64bit.asm:742: error: symbol `vmovdqa64' redefined\nroutines-x86-64bit.asm:742: error: parser: instruction expected\nroutines-x86-64bit.asm:753: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:753: error: parser: instruction expected\nroutines-x86-64bit.asm:755: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:755: error: parser: instruction expected\nroutines-x86-64bit.asm:757: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:757: error: parser: instruction expected\nroutines-x86-64bit.asm:759: error: symbol `vmovdqu64' redefined\nroutines-x86-64bit.asm:759: error: parser: instruction expected\nmake: *** [bandwidth64] Error 1\n",
  "stderr_lines": [
    "ar: creating OOC64.a",
    "routines-x86-64bit.asm:723: error: parser: instruction expected",
    "routines-x86-64bit.asm:725: error: symbol `vmovdqu64' redefined",
    "routines-x86-64bit.asm:725: error: parser: instruction expected",
    "routines-x86-64bit.asm:727: error: symbol `vmovdqu64' redefined",
    "routines-x86-64bit.asm:727: error: parser: instruction expected",
    "routines-x86-64bit.asm:729: error: symbol `vmovdqu64' redefined",
    "routines-x86-64bit.asm:729: error: parser: instruction expected",
    "routines-x86-64bit.asm:739: error: parser: instruction expected",
    "routines-x86-64bit.asm:740: error: symbol `vmovdqa64' redefined",
    "routines-x86-64bit.asm:740: error: parser: instruction expected",
    "routines-x86-64bit.asm:741: error: symbol `vmovdqa64' redefined",
    "routines-x86-64bit.asm:741: error: parser: instruction expected",
    "routines-x86-64bit.asm:742: error: symbol `vmovdqa64' redefined",
    "routines-x86-64bit.asm:742: error: parser: instruction expected",
    "routines-x86-64bit.asm:753: error: symbol `vmovdqu64' redefined",
    "routines-x86-64bit.asm:753: error: parser: instruction expected",
    "routines-x86-64bit.asm:755: error: symbol `vmovdqu64' redefined",
    "routines-x86-64bit.asm:755: error: parser: instruction expected",
    "routines-x86-64bit.asm:757: error: symbol `vmovdqu64' redefined",
    "routines-x86-64bit.asm:757: error: parser: instruction expected",
    "routines-x86-64bit.asm:759: error: symbol `vmovdqu64' redefined",
    "routines-x86-64bit.asm:759: error: parser: instruction expected",
    "make: *** [bandwidth64] Error 1"
  ],
  "stdout": "( cd OOC; make OOC64.a )\nmake[1]: Entering directory `/home/ec2-user/bandwidth-1.7/OOC'\ngcc -Wall -m64 -g -Wno-conditional-type-mismatch -c ObjectOrientedC.c Object.c MutableImage.c Double.c Int.c Array.c MutableArray.c SimpleGraphing.c String.c MutableString.c Font.c FontBuiltin.c\nar rvs OOC64.a ObjectOrientedC.o Object.o MutableImage.o Double.o Int.o Array.o MutableArray.o SimpleGraphing.o String.o MutableString.o Font.o FontBuiltin.o\na - ObjectOrientedC.o\na - Object.o\na - MutableImage.o\na - Double.o\na - Int.o\na - Array.o\na - MutableArray.o\na - SimpleGraphing.o\na - String.o\na - MutableString.o\na - Font.o\na - FontBuiltin.o\nmake[1]: Leaving directory `/home/ec2-user/bandwidth-1.7/OOC'\nnasm -f elf64 utility-x86-64bit.asm -o utility-x86-64bit.o\nnasm -f elf64 routines-x86-64bit.asm -o routines-x86-64bit.o\n",
  "stdout_lines": [
    "( cd OOC; make OOC64.a )",
    "make[1]: Entering directory `/home/ec2-user/bandwidth-1.7/OOC'",
    "gcc -Wall -m64 -g -Wno-conditional-type-mismatch -c ObjectOrientedC.c Object.c MutableImage.c Double.c Int.c Array.c MutableArray.c SimpleGraphing.c String.c MutableString.c Font.c FontBuiltin.c",
    "ar rvs OOC64.a ObjectOrientedC.o Object.o MutableImage.o Double.o Int.o Array.o MutableArray.o SimpleGraphing.o String.o MutableString.o Font.o FontBuiltin.o",
    "a - ObjectOrientedC.o",
    "a - Object.o",
    "a - MutableImage.o",
    "a - Double.o",
    "a - Int.o",
    "a - Array.o",
    "a - MutableArray.o",
    "a - SimpleGraphing.o",
    "a - String.o",
    "a - MutableString.o",
    "a - Font.o",
    "a - FontBuiltin.o",
    "make[1]: Leaving directory `/home/ec2-user/bandwidth-1.7/OOC'",
    "nasm -f elf64 utility-x86-64bit.asm -o utility-x86-64bit.o",
    "nasm -f elf64 routines-x86-64bit.asm -o routines-x86-64bit.o"
  ]
}

Investigate using Batch and AWS spot instances

Should be able to increase automation and greatly decrease cost.

Move terraform and ansible tooling into infrastructure repo

We have a variety of tooling built in terraform for deploying resources in AWS and Azure, as well as some Ansible config for creating various resources. Unfortunately these resources are scattered throughout the PDK (with lots of duplication), and some private repos. It would be useful to have generic versions stored here that we could use for testing/benchmarking, and point users to in order to make it easier to get started deploying Pilosa cluster on the various clouds.

Update files to ensure version compatibility with terraform and ansible

Add required versions and update files as necessary to support current version of Terraform and Ansible.

k8 and ks3

Any plans to support kubernetes ?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble