btargac / excel-parser-processor

Automate downloads from Excel files in seconds. Simply does the tedious, repetitive operations for rows of Excel files and reports the results. It downloads files from URL(s) in column A, if a new filename is provided at column B it will rename before saving. It will even create sub folders if column C is filled with a valid folder name.

Excel Parser Processor

A Desktop app for processing all rows of Excel files

Simply generates an array of items from the rows of an Excel file and does the repetitive tedious operations step by step till every item of the array is processed. For example downloads all the assets from the URLs from column A in an Excel file.

How to use

You can download the latest release for your operating system or build it yourself (see Development).

Just select or drag & drop an Excel file, then select the output folder for the downloaded images or files. All the items in the Excel file will be downloaded into the selected folder, and you will be notified about the state of ongoing progress.

Sample Excel file structure

	A (resource to download)	B (new filename if used)	C (subfolder name)
1	https://www.buraktargac.com/sample_image.gif	optional-sample-file-name	optional-sub-folder-name
2	https://www.buraktargac.com/sample_image.png	optional-sample-file-name	optional-sub-folder-name
3	https://www.buraktargac.com/sample_image.jpg
.	...
.	...
n	Asset URL (any type of file image, text, pdf etc.)

Currently, there is no limit for n, I tested with 4000 items and unless your IP is banned from the publisher there is no problem to download as much as you can.

Demo

Development

You need to have Node.js installed on your computer in order to develop & build this app.

$ git clone https://github.com/btargac/excel-parser-processor.git
$ cd excel-parser-processor
$ npm install
$ npm run build
$ npm start

If you are changing the view or renderer related things, you can use Webpack's watch feature with

$ npm run start-renderer-dev

After running this command, you'll see a webpack process watching your files after a new renderer.bundle.js is generated you can refresh the Excel parser processor app window with cmd + R or ctrl + R depending on your system.

To generate binaries on your computer after your development is completed, you can run;

$ npm run dist

This will add binaries under /release folder on your project folder.

/release folder is ignored at the repository. Github Actions will be building the binaries after your branch is merged with master.

Contributors

This project exists thanks to all the people who contribute. [Code of Conduct].

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

License

MIT © Burak Targaç

excel-parser-processor's Issues

Workaround for sites banning ip?

Very nice tool, tested on some sites and it works wonderfully

However, one of my target site banned me immediately after 80 requests

Any workaround besides VPN, proxies etc?

Can we set the scrapping speed? It's ok for me to scrap slower

How to login to Azure AD ?

How to login to the web if it is requires Azure AD with Multi factor authen?

add analytics support to track usage and such statistics

'electron-google-analytics' package from npm might be a perfect fit for this issue, remember not to set google analytics key, secret or token statically. They can be set on the compile time via webpack define plugin etc.

exception empty excel

gives an error in an empty excel!

can't application restart

add Issue templates

issue templates will force people to open more readable and standart issues

npm run build script fails on Windows platform

> NODE_ENV=production PROCESS_TYPE=main webpack --config webpack.prod.js

'NODE_ENV' is not recognized as an internal or external command,
operable program or batch file.

bugsnag or some similar error tracking mechanism should be added

In-app Navigation and Electron.js Version

Summary:

While the Excel Parser Desktop Application uses secure web preferences, it does not use event listeners that prevent in-app navigation. Moreover, the application can benefit from an update to the underlying Electron.js version.

Platform(s) Affected:

MacOS, Windows, Linux

Steps To Reproduce:

1. Open the Excel Parser Desktop Application.
2. From the “View” menu, choose “Toggle Developer Tools”.
3. [In-app Navigation] Within the console, enter window.location=”https://attacker.com/”. The application window navigates to the third-party site.
4. [Alt. in-app navigation] Alternatively, within the console, enter window.open(“https://attacker.com/”). The application opens a new window with the third-party domain.
5. [Web Preferences] While the app disables nodeIntegration and enables contextIsolation, it does not enable sandbox. These features can be taken care of by the defaults of the latest Electron.js version.
6. [Electron.js Version] Finally, the current version of Excel Parser depends on Electron v17 which is vulnerable to numerous CVEs. [Example] The app can benefit from an update to the framework version that fixes numerous security issues. [Link]

--
Mir Masood Ali, PhD student, University of Illinois at Chicago
Mohammad Ghasemisharif, PhD Candidate, University of Illinois at Chicago
Chris Kanich, Associate Professor, University of Illinois at Chicago
Jason Polakis, Associate Professor, University of Illinois at Chicago

a javascript error occured in the main process

I am getting this problem in some photos.anyone knows ?

Contributing guideline should be created

Allow Image URLs without .JPG/.PNG Suffixes

I have been searching for well over a week for an application that does precisely what this wonderful application can do. However, I ran into one insurmountable obstacle: the website from which I am currently attempting to batch download images (of historical newspapers, journals, magazines, etc.) obscures the absolute path to the images in question. Thus, my image URLs conform to the following format:

http://digitale.bnc.roma.sbn.it/tecadigitale/img/giornale/TO00185283/1880/unico/00000001/original

It would be super if the developers of this application could allow for image URLs such as this one that do not feature a .jpg/.png file extension. As it currently stands, my URLs simply error out. I have tried to use other batch downloaders, and they successfully download these images, but none of them allow for filenames or folder names, which is what makes this application highly attractive.

Everything else about this application is incredible, and I commend the developers for their very good work.

Thank you for your consideration.

create CONTRIBUTING.md

contribution guideline is needed

show the ongoing progress on the client side

Now the percentage or currently downloaded image name is not shown on the renderer process, send IPC messages to renderer process and show on the DOM elements via events and message datas

Brew implementation

Add brew install capability to make cli addicts happier :)

JavaScript Linter should be added to keep code styling consistent.

ESlint with Prettier combination could be a good fit

Rename from next coloumn function is needed

Rename from next coloumn function is needed, as the program goest through excel rows and grabs the file from the link, it would be a great addition to rename the file with a new name described on the next coloumn or any place. If left blank program can still stick with the original file name,

Thanks all for the great program!

Add codetriage badge to readme file

should be added to README.md file

absent documentation

how to use

app fails on http requests when working behind proxy

migrate to net module of electron ( Issue HTTP/HTTPS requests using Chromium's native networking library ) to work proper with system proxy instead of node-fetch module

Nothing happens after file select

I downloaded the windows version 1.3.0. But when I have selected the xls nothing more happens. Tested both drag&drop and using browse.

Expected to get to choose a folder for download. Not showing. No images downloads.

What can be the problem? my column A is Url, column B is filename.

add Pull request template

PR's will be standardized with the new template