GithubHelp home page GithubHelp logo

514-labs / moose Goto Github PK

View Code? Open in Web Editor NEW
24.0 8.0 6.0 72 MB

The developer framework for your data & analytics stack

Home Page: https://www.moosejs.com

License: MIT License

JavaScript 5.23% CSS 0.75% TypeScript 39.58% Shell 1.28% Rust 39.58% MDX 12.47% Dockerfile 0.20% Python 0.92%
data dataengineering deployment framework insights metrics analytics

moose's Introduction

moose logo

MooseJS

NPM Version Moose Community Docs MIT license

MooseJS is an open-source developer framework for your data & analytics stack. Moose is intended for any application where collecting data, processing data, and extracting insights from that data is at the heart of the application. Moose tries to take the decades-old best practices of frontend and backend developer frameworks, and bring them to the world of developing data-driven applications.

Highlights

  • Abstract away middleware and infrastructure complexity: expertise in the data engineering ecosystem not required. MooseJS automatically configures your data stack based on the business logic of your code.
  • Developer-first: Develop in languages you know, in the IDE of your choice. Git-based based workflows for version control and change management. Plug into CI/CD and automated testing workflows. Iterate quickly with local development, then scale to the cloud
  • Hot-reload local dev server: run your whole data/analytics stack locally, and see the impact of changes in real time as you edit code - just like developing a web application
  • Auto-managed migrations: Moose automatically manages schema evolution across your data stack with "Data Change Management", so you don’t break upstream and downstream dependencies
  • End-to-end typed: Moose types all your components from data capture through consumption, so you catch issues at build time, instead of in production
  • Data product APIs: Moose automatically creates API endpoints for your data products, and generates SDKs in the languages of your choice for easy discovery and consumption

Getting started

To learn more and/or get started using Moose, check out the documentation: https://docs.moosejs.com/

Alpha release

Moose is still in Alpha, and is not yet recommended for critical production use cases. That being said, we’d love for you to get your hands on it and try it out. Moose is in active development, and we’re looking for feedback and insights on how to harden and improve, as we build towards the Beta release. You can reach us at [email protected] or in the Moose developer community below.

Community

You can join the Moose community on Slack.

Here you can get together with other Moose developers, ask questions, give feedback, make feature requests, and interact directly with Moose maintainers.

Contributing

We welcome contributions to Moose! Please check out the contribution guidelines.

Made by 514

Our mission at fiveonefour is to bring incredible developer experiences to the data stack. If you’re interested in enterprise solutions, commercial support, or design partnerships, then we’d love to chat with you.

moose's People

Contributors

03cranec avatar asl677 avatar callicles avatar camelcasedaditya avatar cjus avatar dandevries11 avatar datguyjonathan avatar dependabot[bot] avatar georgevanderson avatar oatsandsugar avatar okane16 avatar phisgr avatar tg339 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

nic mcginleyr1 georgevanderson oatsandsugar

moose's Issues

[POL-73] [POL-72] Figure out "migrations" for topics

This should probably include defining a schema for the red panda schema registry to enable data quality enforcement. As I've dug into this it seems like protobufs will be a requirement to be able to enforce schemas in the registry. This is something we've talked about in the past. Given that we're hoping for multi language support, it may make sense to leverage ~~~~~~protobufs~~~~~~ avro as the cross-language serialization it may make sense to make avro easy to work with.

From SyncLinear.com | POL-72

From SyncLinear.com | POL-72

Connection to Clickhouse on start up is brittle

When I spin up dev I get 1/2 times:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Network(hyper::Error(Io, Os { code: 54, kind: ConnectionReset, message: "Connection reset by peer" }))', src/infrastructure/console.rs:38:81
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[1]    99364 abort      ../igloo-stack/apps/igloo-kit-cli/target/debug/igloo-cli dev

Looks like we need to be more tolerant to network resets

[POL-76] Support Prisma as our OLAP ORM

I'm trying to flesh out whether we should create an ORM as part of the igloo framework. It seems like there would benefits both for the developer experience when using the framework but could also serve as an abstraction layer over different databases and stream systems.

It could then integrate nicely into the broader flows paradigms and possibly be used in conjunction with orchestration frameworks.

After exploring this topic we're concluding that contibuting a clickhouse connector to Primsa is like the best path forward. We're excited about the way they do migrations. We'll need to investigate more to understand the LOE here but from the preliminary investigation it seems feasible.

From what I can gather, we'll need DMMF support, type mapping for clickhouse supported types and likely some tweaks to the migrations given that alter statements are different in clickhouse than they are in other systems

From SyncLinear.com | POL-76

Use docker-compose instead of manually stitching docker run together

Right now we have a series of docker run command with a network setup in between.

We are managing all that and making the glue between the different element when we could generate a docker-compose.yaml file in the project directory and use all the optimizations from there when spinning up containers.

docker-compose is shipped with docker, so that should not add another dependency.

That should simplify our code and reduce the number of bugs.

Latest bug I spotted is that if the config of a container changes through a new version of the CLI, the containers fail to boot because we are only creating containers once and then running them. But if the configuration changes, we are not updating it. This is logic that is embedded in docker-compose

sfully validated clickhouse docker container
        Failed to run console container: docker: Error response from daemon: Conflict. The container name "/console-1" is already in use by container "b2cdf08f387864fb94d8546a5767391ef08dd89f7b5813afc51e685d1ff129f6". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

        Failed to run console container: docker: Error response from daemon: Conflict. The container name "/console-1" is already in use by container "b2cdf08f387864fb94d8546a5767391ef08dd89f7b5813afc51e685d1ff129f6". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

        Failed to validate red panda cluster

Fix framework console production build 

./src/app/constants.ts:1:1
#21 19.10 moose-console:build: Type error: 'constants.ts' cannot be compiled under '--isolatedModules' because it is considered a global script file. Add an import, export, or an empty 'export {}' statement to make it a module.
#21 19.10 moose-console:build: 
#21 19.10 moose-console:build: > 1 |
#21 19.10 moose-console:build:     | ^
#21 19.15 moose-console:build:  ELIFECYCLE  Command failed with exit code 1.
#21 19.17 moose-console:build: ERROR: command finished with error: command (/app/apps/moose-console) /usr/local/bin/pnpm run build exited (1)
#21 19.18 moose-console#build: command (/app/apps/moose-console) /usr/local/bin/pnpm run build exited (1)

And Add some CI to make sure we don't break this inadventendly

[POL-78] Support interface based ingestion abstraction

We've explored leveraging an ORM to create the needed abstraction for the ingestion point. We believe that an ORM DSL based approach, like Prisma Schema Language is the best past forward where the ingestion point would take in a ORM generated type as it's input and enable the user to easily create an ingestion point.

With that in mind, though, we'd like to enable users to create ingestion points quickly. To do so, we will support creating an ingestion point and passing it a typescript interface as a generic type or function parameter (approach TBD).

The hope is for the user to be able to run this command and then have the ingestion point configured for them. We need to support:

  1. creating the table in the database;
  2. creating the topic in red panda
  3. creating the sql statement to ensure that the topic is constantly pushing data to the table
  4. creating the typescript code for the developer to add parameters
$ igloo add ingestion-point --name=user-event -a=timestamp:utc -a=packet:json -a=type:enum

The hardest problem will be enabling the developer to write code against typescript files and automatically update the migration. Alternatively, we can force the developer to define their parameters through the CLI or to manually update their migration scripts if they decide to write parameters manually.

From SyncLinear.com | POL-78

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.