yobulkdev / yobulkdev

🔥 🔥 🔥Open Source & AI driven Data Onboarding Platform:Free flatfile.com alternative

YoBulk
⭐️ Open Source Alternative To Flatfile.com ⭐️

A Data exchange platform for all businesses that run on CSV files. Create YoBulk CSV Import button and make CSV importing 10x faster.

Website • Twitter • Slack

See other languages »

User Installation

• Docker

Docker

Option:1 Through Docker Compose

The docker compose brings both YoBulk Image and Mongodb Image. We provide docker-compose.yml file under yobulkdev code repository.

git clone https://github.com/yobulkdev/yobulkdev.git
cd yobulkdev 
docker-compose up -d

• To make use of YoBulk Open AI feature, please follow the guidelines as below.

Please open the docker-compose.yml file and update the OPENAI_SECRET_KEY variable with the key generated from OpenAI API site.

The YoBulk dashboard is ready to go at http://localhost:5050/.

Option:2 Through Docker Run

If you are a Docker user, you may try this way!

• Prerequisites:You should have installed mongodb on your local machine and it's running.

docker run --rm -it -p 5050:5050/tcp  yobulk/yobulk

• To make use of YoBulk Open AI feature, please follow the guidelines as below.

Please pass the Open AI key generated from OpenAI API site in --env to explore YoBulk AI features.

Example:

docker run --rm -it -p 5050:5050/tcp  --env="OPENAI_SECRET_KEY=****" yobulk/yobulk

YoBulk dashboard is ready to go at http://localhost:5050/.

Building locally.

If you are a developer and want to build the system locally, you should have Mongodb running locally.

YoBulk is a fullstack next.js application which uses Mongodb as it's primary Database.

git clone https://github.com/yobulkdev/yobulkdev
cd yobulkdev
yarn install
yarn run dev

• To make use of YoBulk Open AI feature, please follow the guidelines as below.

Please update OPENAI_SECRET_KEY variable in .env file with the Open AI key generated from OpenAI API site

The dashboard is ready to view at http://localhost:5050/.

YoBulk Cloud

YoBulk Cloud is the fastest and most reliable way to run YoBulk. You can get started with free credits in minutes.

Sign up for YoBulk Cloud.

Quick start

Please view the documentation to explore CSV importing flow.

Roadmap

• YoBulk Roadmap ReadMe

Features

Supported Features

• ⚡  No code Template and Import button creation
• ⚡  Smart auto matching between the CSV column and Template column
• ⚡  Custom validation rule setting against a field
• ⚡  Delightful Data review experience
• ⚡  Scalable through streaming.You can import a CSV with size of 1GB
• ⚡  GPT3 Integration
• ⚡  Template creation through JSON
• ⚡  Regex creation and addition in custom validation
• ⚡  Collaboration & workspace creation.
• ⚡  AI based auto suggestion for correcting errors

Coming Soon:

• ⚡  Custom LLM models for Auto suggestion and Auto Cleaning of CSV errors
• ⚡  Data mapping decission knowledge graph for different sectors

Demos

NoCode Template Creation Demo

No_code_Template.mp4

All DataType Validation Demo

All_DataType_Validation.mp4

String Validation Demo

yo_string_validation.mp4

GPT Demo

GPT.Demo.mp4

Why are we building this?

More than 70% of the business data is shared in the form of CSVs and Excel files. Less than 10% have truly adopted API integrations for data exchange.

As a developer and product manager, we have gone through the pain of building a scalable CSV importer. Everyday thousands of developers go through the same pain and hundreds of companies have to get enormous amount of spreadsheet and CSV data into a database.

YoBulk is building an OSS standard for CSV and flatfile data onboarding where it can help you to build a data importing solution in-house.

Enterprises like Flatfile.com are addressing this problem but it has constraints like weak data security and high pricing. While developers want an embeddable/scalable CSV importer for SaaS, the truth is that, it is not possible to come up with an "one-size fits all solution" within such fragmented market.

All these restrictions necessitate our mission, to make it possible for anyone to create an data import solution that suits their needs well.

To Developers, we would like to offer a "bring your own validations"(BYOV) and "bring your own database"(BYOD) experience. You can design and modify validations rules on your own way using YoBulk.

To enterprises , YoBulk is dedicated to bring data security. Moreover, you have 100% control of your data armed with long-term maintainability.

To be honest, we do not claim to outperform flatfile.com in terms of functionality and design, at least for now. Besides, our priority is to add more functionality at the moment. We would like to cultivate a community to democratise the knowledge and bring all CSV and flatfile open source projects under a single framework.

Read more Open vs Closed Data Onboarding

Our Mission

Our mission is to provide the most powerful flatfile(CSV,Spreadsheet) importer which is open source to every single internet business in the world. An AI first data exchange for businesses to share flatfiles where you are in-charge of your customer’s data onboarding, validation & transformation. We stongly support Anderz's vision.

Our Vision

YoBulk's vision is to create an AI first knowledge graph for carrying out over billions of data mapping decisions while exchanging data between businesses across various sectors through flat files such as CSV and Excel.

Support

If you have any questions or problems, please visit our GitHub discussions page, and we'll try to help you as soon as possible.

https://github.com/yobulkdev/yobulkdev/discussions

License

YoBulk is an open-source software with the AGPL 3.0 license.

Contributors

We welcome contributions to YoBulk.

yobulkdev's Issues

[Enhancement]:While creating a template ,provide an example of an accepted data field.

Example:
While creating the template, if the data is marked as country, then give an example as US or IND.

SaaS developer will be given sample regex keywords as references in Document to refer.

YoBulk should support column hooks to do custom validation at cell level.

Example: Column hook Input
[
{ value: "[email protected]", index: 0 },
{ value: "[email protected]", index: 1 },
{ value: "[email protected]", index: 2 },
{ value: "[email protected]", index: 3 },
];

Column Hook Output:
[
{
index: 0,
value: "[email protected]",
info: [
{
message: "Automatically corrected email domain",
level: "info", // should be "info", "warning" or "error"
},
],
},
// we can omit cells that don't need any changes or messages
{
index: 2,
// Don't need to supply value if we're not changing it
info: [
{
message: "Email not found",
level: "error",
},
],
},
];

Updation of Readme file to set the user flow and story correctly

1.Banner diagram is not conveying the solution properly.
2.Remove unnecessary data and information.

End User can able to invite other users as a collaborator during review state.

End user can able to retrieve any data if he wants to resume the review work later.

The client SDK and Platform SDK should work for Angular and Vue applications.

[Enhancement]:SaaS developer can able to add a Regex in template validation form.

SaaS developer can able to add warning type for a specific validator field.

User will be given a heads up on the max records can be imported through a single CSV file.

Specify the maximum number of records that a single file can import. This value does not include the headers of the file.

If the number of rows in the file exceed this number, the user will be warned and the excess rows will be dropped.

Progress bar becomes 100 percent and waits for couple of seconds

As the response for the api to upload filestream is resolved after the write stream finishes, the upload progress bar waits at 100pc progress.

SaaS developer can able to add cross column/field validation<error messaging is depending on two or more fields>

SaaS developer can able to select a specific date format against which the column validation can be done

Right now YoBulk support around 8 types of date validation types.

YoBulk will support taking the flatfile data in json object without loading a CSV

This allows you to specify the data being uploaded, bypassing the CSV upload stage.
As an array of objects, where each object represents a row.
[
{
name: "Jack",
email_address: "[email protected]",
},
{
name: "Jill",
email_address: "[email protected]",
},
];

YoBulk will enable SaaS developer/Data acceptor to check the GDPR compliance of the data which needs to be imported.

YoBulk SaaS developer should able to create a workspace and attach specific user to that workspace.

A single workspace can be shared between multiple end user.
A workspace can have multiple template and multiple CSV ID attached to it.

Template should support both unique and required options.

Required:Validation error for any blank value.
Unique:Will throw an error for any duplicate value.

User will be shown a count of total cells formatted or corrected after completion of Import job

User/Data provider(Customer of SaaS) can able to access a dashboard.

The User/Data provider(Customer of SaaS) dashboard should show the following things.

User will be given recommendation to auto correct a fix across a column or field.

customers can export data to Excel while keeping Yobulk’s error highlighting and error messages.

SaaS developer can able to configure an endpoint url as webhook for sending the validated data.

Addition of codepen/ Glitc/codesandbox editor to simulate the embedding button usecase.

Post creation of template ID, SaaS developer should be shown a way to create a button creation through codesandbox similar to flatfile.

Reference:https://codesandbox.io/s/portal-example-javascript-n1crwg?file=/src/index.js

End User should be able to filter the errors based on the Types.

The Importer will show a filter option to select the errors based on the column type.
It will help them to work on specific error and doing quick fix.

[Enhancement] Show the template's expected fields when the user is uploading a CSV for first time

SaaS developer can able to provide the corrected imported file in JSON format to his customer.

Customer may convert the json to CSV at his end.

An Admin Dashboard is required for SaaS developer to get an Track of the Upload.

The Admin Dashboard will show the following things:
1.CSV file Name
2.Workspace ID
3.Date and Time of Uploads.
4.Number of Records uploaded.
5.Number of records skipped.
6.Template ID Attached to Each CSV

SaaS developer can able to send a link to their customer.Clicking the link will start a standalone onboard process.

User will be allowed to download all the error as a CSV.User can resubmit that CSV again.

End user will be shown the row number in review state.

YoBulk will support full validation in browser only if the end user is not ready to share data with YoBulk

Lot of core features will not be available in this mode of working.

SaaS developer can able to create a template from code or JSON data.

Boolean column value is not visible in data grid

The react-ag-grid is used in the dataviewer. If the column value is true or false, then the value is not visible in the column. The idea could be to implement a checkbox method to tackle this issue

SaaS developer can able to add custom themes to importer button and complete importer UI including color,font,progress bar,color code for errors.

User will be given an option to see only error records

A toggle button will be given to get a view of only error records.
This will help in resolving errors quickly.

[Enhancement] SaaS developer will be provided with a catalog to auto create linkedin,Hubspot templates.

User will be shown a AI based recommendation option if configured in Template.

This can be used to give the user additional context on what issues they should resolve on this step.

SaaS developer can able to create a template by uploading a sample CSV from from his customer /Data provider.

UI Flow should change to Data upload->Matched->ReviewTable->Submit data->Show Progress of Importing-->Available in UI for download or API endpoint to consume

Flow will go back from submitted data to review table state if the submission fails.

After review state and in post submission state ,user will be shown a progress bar once the data ready in UI to sent successfully to an endpoint url

User and SaaS developer's Admin dashboard can able to retrieve the original CSV file uploaded before correction for any future comparision

Note:Doing the transformation during streaming may need to change.

YoBulk on prem deployment should support SQL database instead of MongoDB.

A new better UI for mapping or matching has to be designed.

[Enhancement]Confirm the first row as the field header from the user after uploading a CSV file.

YoBulk should support other flatfile formats importing like XLS.

Webhook implementation for uploading the cleansed csvs

Once the user cleanses the csv, there is a download button which downloads the whole file. Is it a good idea to give a webhook or an api input where we can stream the data into.

End user can able to see only those records where modification is done in review state.

Removing standalone Importer functionality from YoBulk's Data Engine

Standalone Importing is an end client/user action.It should be decoupled from the YoBulk Data engine.

The matching or mapper UI should exclude the column from the column list which is already matched or mapped.

SaaS developer can able to add different color codes for any specific field in a template.

YoBulk data engine / backend should know the status of the UI through events.The client SDK needs to be redesigned.