GithubHelp home page GithubHelp logo

pipedreamhq / pipedream Goto Github PK

View Code? Open in Web Editor NEW
8.3K 278.0 5.3K 264.72 MB

Connect APIs, remarkably fast. Free for developers.

Home Page: https://pipedream.com

License: Other

JavaScript 89.32% Shell 0.01% TypeScript 5.28% Jupyter Notebook 0.12% Python 0.53% CSS 0.01% MDX 4.72%
eventsourcing event-sourcing automation workflow workflows apis data-flow integration-flow cli low-code

pipedream's Introduction

pipedream

Pipedream is an integration platform for developers.

Pipedream provides a free, hosted platform for connecting apps and developing event-driven automations. The platform has over 1,000 fully-integrated applications, so you can use pre-built components to quickly send messages to Slack, add a new row to Google Sheets, and more. You can also run any Node.js, Python, Golang, or Bash code when you need custom logic. Pipedream has demonstrated SOC 2 compliance and can provide a SOC 2 Type 2 report upon request (please email [email protected]).


HTTP trigger + step selection menu

This repo contains:

This README explains the key features of the platform and how to get started.

To get support, please visit https://pipedream.com/support.

Key Features

  • Workflows - Workflows run automations. Workflows are sequences of steps - pre-built actions or custom Node.js, Python, Golang, or Bash code - triggered by an event (HTTP request, timer, when a new row is added to a Google Sheets, and more).
  • Event Sources - Sources trigger workflows. They emit events from services like GitHub, Slack, Airtable, RSS and more. When you want to run a workflow when an event happens in any third-party app, you're using an event source.
  • Actions - Actions are pre-built code steps that you can use in a workflow to perform common operations across Pipedream's 1,000+ API integrations. For example, you can use actions to send email, add a row to a Google Sheet, and more.
  • Custom code - Most integrations require custom logic. Code is often the best way to express that logic, so Pipedream allows you to run any Node.js, Python, Golang, or Bash code. You can import any package from the languages' package managers, connect to any Pipedream connected app, and more. Pipedream is "low-code" in the best way: you can use pre-built components when you're performing common actions, but you can write custom code when you need to.
  • Destinations - Deliver events asynchronously to common destinations like Amazon S3, Snowflake, HTTP and email.
  • Free - No fees for individual developers (see limits)

Demo

Click the image below to watch a brief demo on YouTube.


Pipedream demo static image

Workflows

Workflows are sequences of linear steps triggered by an event (like an HTTP request, or when a new row is added to a Google sheet). You can quickly develop complex automations using workflows and connect to any of our 1,000+ integrated apps.

See our workflow quickstart to get started.

Event Sources

Event Sources watch for new data from services like GitHub, Slack, Airtable, RSS and more. When a source finds a new event, it emits it, triggering any linked workflows.

You can also consume events emitted by sources using Pipedream's REST API or a private, real-time SSE stream.

When a pre-built source doesn't exist for your use case, you can build your own. Here is the simplest event source: it exposes an HTTP endpoint you can send any request to, and prints the contents of the request when invoked:

export default {
  name: "http",
  version: "0.0.1",
  props: {
    http: "$.interface.http",
  },
  run(event) {
    console.log(event); // event contains the method, payload, etc.
  },
};

You can find the code for all pre-built sources in the components directory. If you find a bug or want to contribute a feature, see our contribution guide.

Actions

Actions are pre-built code steps that you can use in a workflow to perform common operations across Pipedream's 500+ API integrations. For example, you can use actions to send email, add a row to a Google Sheet, and more.

You can create your own actions, which you can re-use across workflows. You can also publish actions to the entire Pipedream community, making them available for anyone to use.

Here's an action that accepts a name as input and prints it to the workflow's logs:

export default {
  name: "Action Demo",
  description: "This is a demo action",
  key: "action_demo",
  version: "0.0.1",
  type: "action",
  props: {
    name: {
      type: "string",
      label: "Name",
    },
  },
  async run() {
    return `hello ${this.name}!`;
  },
};

You can find the code for all pre-built actions in the components directory. If you find a bug or want to contribute a feature, see our contribution guide.

Custom code

Most integrations require custom logic. Code is often the best way to express that logic, so Pipedream allows you to run custom code in a workflow using:

Node.js Python
Go Bash

You can import any package from the languages' package managers by declaring the imports directly in code. Pipedream will parse and download the necessary dependencies.

// Node.js
import axios from "axios";
# Python
import pandas as pd
// Go
import (
    "fmt"
    pd "github.com/PipedreamHQ/pipedream-go"
)

You can also connect to any Pipedream connected app in custom code steps. For example, you can connect your Slack account and send a message to a channel:

import { WebClient } from "@slack/web-api";

export default defineComponent({
  props: {
    // This creates a connection called "slack" that connects a Slack account.
    slack: {
      type: "app",
      app: "slack",
    },
  },
  async run({ steps, $ }) {
    const web = new WebClient(this.slack.$auth.oauth_access_token);

    return await web.chat.postMessage({
      text: "Hello, world!",
      channel: "#general",
    });
  },
});

Destinations

Destinations, like actions, abstract the connection, batching, and delivery logic required to send events to services like Amazon S3, or targets like HTTP and email.

For example, sending data to an Amazon S3 bucket is as simple as calling $send.s3():

$send.s3({
  bucket: "your-bucket-here",
  prefix: "your-prefix/",
  payload: event.body,
});

Pipedream supports the following destinations:

Contributors

Thank you to everyone who has contributed to the Pipedream codebase. We appreciate you!

Pricing

Pipedream has a generous free tier. You can run sources and workflows for free within the limits of the free tier. If you hit these limits, you can upgrade to one of our paid tiers.

Limits

The Pipedream platform imposes some runtime limits on sources and workflows. Read more about those in our docs.

Found a Bug? Have a Feature to suggest?

Before adding an issue, please search the existing issues or reach out to our team to see if a similar request already exists.

If an issue exists, please add a reaction or add a comment detailing your specific use case.

If an issue doesn't yet exist and you need to create one, please use the issue templates.

Security

You can read about our platform security and privacy here.

If you'd like to report a suspected vulnerability or security issue, or have any questions about the security of the product, please contact our security team at [email protected].

pipedream's People

Contributors

ajrice6713 avatar alysonturing avatar andrewjschuang avatar bush4u avatar celestebancos avatar ctrlaltdylan avatar danhsiung avatar dannyroosevelt avatar dependabot[bot] avatar dylburger avatar feyzullah avatar gtfalcao avatar jcortes avatar js07 avatar jsendo avatar jverce avatar lcaresia avatar luancazarine avatar lucberge avatar malexanderlim avatar masonwoodford avatar michelle0927 avatar mjabali avatar pipedream-component-development avatar psavkar avatar sergioeliot2039 avatar tjk avatar todsac avatar vellames avatar zalmanlew avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pipedream's Issues

Fullscreen mode for app / action search

Right now the list of actions can be quite long for some apps, and I don't have a way to open that list in fullscreen mode like I do for editing code. I'd love to expand the app / action search list so it's easier to see everything in one place.

Unified log / API to retrieve all workflow invocations + errors

I'd like to access a log of all workflow executions (my event history) or errors so I can run analysis on them and observe patterns. The log would need to contain metadata like:

timestamp of execution
duration of workflow execution
success / failure state
workflow id
deployment id
(any other relevant data from steps.trigger.context)
timestamp of error (if error)
error object (if present)

Workflow creation API

I'd like to programmatically create a workflow via API, passing the relevant data (title, description, trigger, steps, etc.) as parameters.

Autocomplete in action forms should work like it does in code steps

In code steps, I'm presented with an auto-complete menu that gives me the property at the next "level" in a given object, e.g. I type steps. and get the properties of steps that I can select from:

Screen Shot 2019-10-30 at 16 20 18

Selecting one of these properties presents yet another "level" of properties, if there are more to show, and I can continue until I select the value I want.

I like this form of auto-complete and would like it in actions forms, as well. I still like the sample values presented next to the values in the "action-style" autocomplete, so ideally I'd like to keep those, too.

[TRIGGER] GMAIL

I'd like to run workflows on Gmail events, e.g. when a new email arrives in my Inbox.

Set rules for what events trigger a workflow

I might enable a large stream of events via webhook, for example, that hit my workflow. But I might only need to process a subset of the event types. I'd like a way to filter those events before my workflow runs via some set of rules.

I can add a code step to perform this filtering today, but my workflow still executes for every event. I'd like my workflow to execute only on the events I care about.

I need a way to replay an event with a modified payload

When my workflow yields an error, sometimes it's because my source data was messy. I want to modify the event payload slightly before replaying that event through my workflow.

Other tools โ€” for example integrator.io โ€” have an optimized flow for this use case.

As a workaround, I can edit the test event for my workflow, add the modified payload there, and send the event through the workflow. But this is time consuming when I get a lot of errors in my workflow.

Regular email summary of workflow executions

I'd like to get a regular email with a summary of my workflow executions, similar to Zapier or other services.

Like the Global Error Workflow, it'd be great if this were a workflow I could modify, so that I could send the summary to Slack or some other destination, or modify the data sent in the summary.

Action: trigger another workflow

I'd like to trigger a workflow from another workflow via a built-in action.

Workaround: I can create an HTTP trigger for the second workflow and send an HTTP request from the first, but this isn't as clean.

Workflow-level environment variables

e.g. via the CLI, if I have a .env file in a local project, I'd like Pipedream to read and apply the values of those environment variables, using them for the relevant workflow.

[APP] Gmail

I'd like to run Gmail actions as workflow steps, e.g. I'd like to send a new email or attach a label to an email using Pipedream.

Language Support: TypeScript

I'd like to author TypeScript code steps on Pipedream, similar to how I can choose to "Run Node.js code" today.

Deploy / Update Workflows via Git, API, CLI

I want to write and test workflows on my local machine, storing the workflow's code in my existing version control system.

I also want to programmatically deploy workflows using a CLI, for example using a command like pipedream deploy. This way, I can deploy workflows locally or as part of my CI/CD pipeline.

Related: #1139

Rename a workflow while forking

Is your feature request related to a problem? Please describe.
I want to rename a workflow during the forking process.

Describe the solution you'd like
After hitting "fork this workflow" I'd expect to see a modal with a text field that allows me to either enter a new name or modify the existing workflow name (potentially with "copy" appended to the end or something similar.

Do you have a workaround?
Forking then renaming afterward.

Comparable features in other tools?
Yea probably somewhere, but I don't have a reference handy : )

Additional context
Add any other context or screenshots about the feature request here.

When executing :w to save in Vim mode, the app loses focus on the step I'm currently editing

Describe the bug

When executing :w to save in Vim mode, the app loses focus on the step I'm currently editing. I have to click into the step again to start editing.

To Reproduce
Steps to reproduce the behavior:

  1. Enable Vim mode in Settings
  2. Add a new Node.js code step
  3. Write some code, press :w
  4. Editor loses focus

Expected behavior

Ideally, I'd like to keep the browser's focus in the editor for the current step.

Desktop (please complete the following information):

  • OS: macOS
  • Browser: Chrome
  • Version 78.0.3904.70

Loops in workflows

I'd like to run a set of steps or actions within a loop construct. For example, I might receive an array of 100 events in my HTTP payload, and I need to run an action on each. I can do this with code, but I have no way to loop through the elements of this array using Pipedream's actions. I'd like a "loop" action that can handle this use case.

Restrict workflow execution via API key

For HTTP or Email triggers, anyone who knows the destination URL or address of the workflow can send events to it. I can manually add validation in a code step to ensure workflow events come from some source (e.g. only run the workflow when a header is set to a specific value), but I'd like a built-in way to restrict how a workflow is triggered, via API key or some other mechanism.

Support for branching (if / then) operation

Currently, workflows are a linear sequence of steps. I'd like support to branch a workflow in a conditional manner (if condition is true, run this step, else run this other step).

Read / write to shared state across workflows

I need a way to store data in Pipedream in one workflow and read it in another. Currently, data saved using $checkpoint is scoped to a workflow, so I don't have an easy way to share that data across workflows.

[TRIGGER] Slack

I'd like to see common Slack events as triggers, for example I want to trigger a workflow every time a new message is posted to a channel.

Support for parallel execution of tasks (I want to spawn N instances of a step)

Currently, workflows are a linear sequence of steps. I'd like a way to spawn N instances of a step, each of which operates on a subset of the input data.

For example, if I have 100 URLs and want to make HTTP requests to each, I have to write a Node.js code step using something like Promise.all to make those requests in parallel. I'd prefer to use the Send HTTP Request action, feeding in each of the 100 URLs to that action, and run them in parallel.

I also need a way to process the results of these parallel operations once each of the tasks has completed.

Think: map / reduce semantics, or something like JavaScripts Promise.all.

Language Support: Golang

Similar to how I can "Run Node.js code" in a step, I'd like to "Run Golang code" so I can write Go on Pipedream.

Sorting of env variables

Because my workflows are getting bigger and bigger, I ended up with lot of environment variables (API keys, external services end-points urls, and so on).

I would suggest, within "Settings" section, to have such env variables sorted by name (ascending and/or descending). Because it is getting really hard to find the right env variable while I'm writing new code, it would be really nice to have an arrow icon (up/down) that implements the sorting, like in the workflows list.

Many thanks!

Possibility to delete all log entries relative to a whole day

When a workflow needs a long debugging stage, it is highly probable that it will accumulate a lot of log entries. At the moment such entries can be deleted only one by one.
It would be nice to have the option to delete all entries realtive to a day (an [x] button near the labels days ago)
pipedream_suggestion

See attached screenshot

Language Support: Python 3

Is your feature request related to a problem? Please describe.

No

Describe the solution you'd like

Just like Pipedream exposes a way to "Run Node.js code", I'd like a way to "Run Python code". I'd expect support for Python 3 only since Python 2 will be EOL in 2020.

Do you have a workaround?

I need to use specific packages like pandas for which there is no npm equivalent, so even though I can write Node.js for certain things, I need Python.

Puppeteer / Playwright support

Puppeteer provides a Node API to drive a Chrome headless browser. This allows you to programmatically visit sites, take screenshots, and more.

PIpedream currently doesn't support Puppeteer, but I'd like to run workflows that use it.

Status page for pipedream.com services

I'd like a status page for pipedream.com services so I can monitor the health of the service and understand if there's a system issue affecting my workflow.

Allow me to select the OAuth scopes I'd like to allow for a specific connected account

Currently, Pipedream requests broad OAuth scopes for many apps so that, upon connecting your account, you're able to perform any of our listed actions.

But sometimes I just want to perform a very narrow task, and allow Pipedream access to a subset of the listed scopes.

Other times, I might want to authorize one step of a workflow with access to one scope, and another step access to another, so I can tightly control what steps have access to what auth. This is especially important if I'm running a third party npm package that interacts with a service, and I want to carefully control the access that code has to my account.

Overall, some mechanism for selecting my own scopes when I authorize a connected account in Pipedream feels like it would solve these use cases, but I'm open to any other ideas.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.