acrosman / salesforce2sql Goto Github PK

The knex library is used for database abstraction. To test those features we need a good mock or library that projects jest support for knex.

My default values aren't being set properly. Defaults for Varchar(255)s are set without being enclosed in single quotes, so they aren't quite working in practice.

Originally posted by @allisonletts in #30 (comment)

Summary
JSForce supports OAuth connections, they should be used.

Is your feature request related to a problem? Please describe.
Using the old security token system is not ideal. OAuth connections would be a better choice for most users.

Solution
Setup Salesforce2Sql to use OAuth connections to Salesforce.

Describe the bug
Picklist values with ' in them cause errors.

To Reproduce

1. Create a picklist value on a Salesforce object that contains a '
2. Connect to the org and select the object.
3. Attempt to generate the database from that object. Error appears in the log that includes:

ER_PARSE_ERROR: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near

Expected behavior
Table to be created successfully.

Summary
There should be a status update during SF API calls cycles to get object list and field details.

Please describe the problem you are trying to solve.
SF calls are slow, and for a big project there are lots of them. The interface is largely useless while they are running and nothing provides an indication it's running.

Proposed Solution
Initially at least greying the screen and switching to a throbber would be better than current state. But really a progress display that provided updates about what phase the API process was in would be much better.

Summary
We need automated testing. Since Jest is SFDC's library of choice seems like the right choice here.

Summary
There are lots of changes in the main branch, and pending, that justify a new release. There are several details that should get wrapped up before that happens.

• Finish first version of search: Merge #40 to fix #10
• Fix #41
• Fix #42
• Update ReadMe file to have screenshots from new interface.
• Ensure 3rd party packages are up-to-date.

Summary
sf_calls.js has several undocumented functions that could benefit from good doc comments.

Summary
I'd like to be able to say "we use case teams" or "we use pricebooks" or other common features and have the tool check more objects for my schema

Please describe the problem you are trying to solve.
I wish I didn't have to try to remember which objects are included within a particular Salesforce feature.

Proposed Solution
Show a list of features to add or remove using checkboxes. Use the checkboxes drive the selected objects for the schema. This could be a "step 1" where "step 2" is customizing with more granularity (the existing detailed selection).

Summary
Add the ability to update a schema based on changes in source org or to add tables.

Please describe the problem you are trying to solve.
Currently the schema save/load process just allows you to regenerate the SQL not update the fields based on changes in Salesforce. It would be a better solution if you were able to tell it to refresh the org objects and fields based on connected org.

Proposed Solution
Once the schema loads from file it should take the user to the Object list, and present a button refresh the objects from Salesforce. When loading and proposing the schema itself the user should be able to indicate (via checkbox?) if they want existing objects refreshed or just to use what was already loaded from file.

To do this well the interface should really be rebuilt so you can easily move forward and back through the process, with controls only showing on the sections that are active.

Increasingly the SF_calls.js module has several other things in it that has nothing to do with calling Salesforce. The code should be refactored to rename that file to be more accurate and potentially separate the actual SF calls into their own file again or some other similar pattern.

Describe the bug
Some issues with data structures that have come up in practice:

• Salesforce checkboxes come over as Bit(3) - expected Bit(1)
• Enum should be changed to String--Talend is not super happy with Enum
• Default values...maybe just shouldn't be set? Is that okay? The nulls are causing more problems than they're solving for me.

Desktop (please complete the following information):

• OS: Windows Server 2019 Standard
• Target Database: MariaDB

Summary
Create a message to display after the database generation completes to show status of each table.

Please describe the problem you are trying to solve.
Currently nothing appears in the Last Response area to let you know it completed.

Proposed Solution
A popup message that announces the results, and a table-by-table break down that can be displayed in the Last Response area.

Issues to closed before the next release:

• #34
• #35
• #48
• #60
• Update packages.

Summary
The ReadMe would be more useful and appealing with a few screenshots of the interface.

Summary
Right now we push all SF decimal field types to float. They should really all be pushed to decimal

Code to update is here.

When loading the schema from a file the recently redesigned interface should advance from the object screen to the schema page on load but does not.

Describe the bug
Knex auto generates index names, meaning that long field names can lead to even longer index names. Salesforce field names can be quite long. This tool then records an error in the log. Fortunately the indexes are added as table alters so the actual table is created just fine, Indexes on the table after the failure are lost.

To Reproduce
This can easily come on up Namespaced tables like the following from Blackthorn:

Error 1059(ER_TOO_LONG_IDENT) creating table: alter table `conference360__Table__c` add index `conference360__table__c_conference360__main_contact_attendee__c_index`(`conference360__Main_Contact_Attendee__c`) - ER_TOO_LONG_IDENT: Identifier name 'conference360__table__c_conference360__main_contact_attendee__c_index' is too long. Full statement: alter table `conference360__Table__c` add index `conference360__table__c_conference360__main_contact_attendee__c_index`(`conference360__Main_Contact_Attendee__c`)

Proposed Solutions

Minimal: Catch the error so all other indexes can complete.
Better: Find a way to replace the generated index name from Knex with something shorter.

Setup

Windows 10
MySQL

Summary
Add a feature to save the generated SQL file for editing, and reuse.

Please describe the problem you are trying to solve.
It could be useful to save the SQL statements to disk not just run them.

Proposed Solution
In the database modal add an option to save the SQL statements to a file (needs a save dialog call like the one being built for #13). Knex can do this without an active database connection so that might be an add-on feature as well.

Describe the bug
After generating SQL into a database the Save SQL button should be active to allow that same SQL to be dropped to disk. But under some conditions it is not.

May be tied to there being an error during the SQL processing with the database. Myabe just be a regression in v0.4.0

Currently the main render.js file has no test coverage. Testing at least the helper functions near the top of the file would be quite useful.

Summary
With the merge of #49 which closed #48, we now have some test coverage. Bu there is only a little test coverage at the moment. Over time we'll work toward more.

This issue is for generally reminding of the need to include more tests. Additional sub-issues and related issues will be added for specific requests. If you'd like to add tests generically, you may want to look at the functions in src/sf_calls.js and src/preferences.js that do not callback to the main window directly (the project already includes rewire to make it easy to access non-exported features of the modules).

There are also no tests currently covering the render JS files, which are fairly extensive as well.

Specific Testing Issues

• #64
• #65
• #66
• #67
• #68
• #69
• #70

Describe the bug
Objects with lots of fields may violate the table row size in the database. We should detect the that behavior and protect against it.

To Reproduce
Create a Salesforce object with a large number of fields, particularly large text fields (not text areas or rich text), and then attempt to generate a table from that object. The log message may contain a failure based on this error:

SQL Error (1118): Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs

Suggested Fix
Catch the error during processing and replace all varchar fields with text fields. That generally will adjust the size as needed and the importance of a varchar over text is rare for most use cases that want to use a tool like this.

Describe the bug

Currently handleLogin in app/render.js pulls the submitted user name from the input form instead of a value from the response. Ideally this would use a value from the main process's response.

Describe the bug
Currency datatype is Decimal (8,2) regardless of SF settings

To Reproduce
Steps to reproduce the behavior:

1. create a Currency field in salesforce (16,2)
2. Use app to create sql
3. Look at currency field length and note datatype Decimal, 8,2 length/precision.

Expected behavior

• Datatype Decimal (or, like, Double would be fine)
• Length 18
• Precision 2

Screenshots

Desktop (please complete the following information):

• OS: Windows Server 2019 Standard

Additional context
Noticed the problem because I was getting out of range issues on several currency fields.

The various IPC calls block testing of several important codes. The project either needs a mock that supports the needed elements for those calls, or needs a library to provide Jest support for electron other ways.

Summary
Allow users to select from the present themes

Please describe the problem you are trying to solve.
Bootswatch is used to provide the dark theme, but there are several other themes in the package, might as well let people access them.

Proposed Solution
Add a preferences pane that allows users to select the theme from the list of options that are installed anyway.

Currently buildFields() is only tested via the buildSchema() tests. To help find and fix errors as new features are added to each stage more details coverage of the helpers would be useful. Following the pattern of testing for buildSchema, we should add tests to cover buildFields in a similar fashion.

Summary
Currently we use a series of object describes, which for large orgs is very slow. A query of FieldDefinitions should allow for pulling of all, or at least most, field descriptions in one query and only 4 or 5 cycles to Salesforce API (instead of hundreds).

Currently the src/preferences.js has no test coverage. It would be helpful to have a test class that at least loads the module and validates that the module structure matches the expected. The first two tests covering sf_calls.js provide a pattern to work from.

Any additional test coverage is also welcome and appreciated.

Summary
If I'm creating a schema to plan a migration, I want to ignore the R/O fields.

Please describe the problem you are trying to solve.
It's hard to tell which fields shouldn't be mapped to, even though it'll throw an error in SF.

Proposed Solution
Options:

• create the column but set the default to ReadOnly or something so that it's scannable from talend
• don't create the column at all

Describe the bug
When you log out of a Salesforce org, the org is still listed on the set of active collections.

To Reproduce

1. Connect to two orgs.
2. Select the first org and click logout.
3. Both orgs are still options for future requests.

Expected behavior
The disconnected org should be removed.

Add a Select All/ Deselect All checkbox for object selection.

After fetching all objects having a checkbox to Select All/ Deselect All to make it easier to grab only needed objects faster.

Summary
Bootstrap 5 has been released and it would be great to update to it (along with Bootswatch) and deal with legacy issues.

• Update references
• Remove Popper.js 1
• Update JS links to use combined.
• Update for removal of .pre-scollable
• Update modal and data classes to include the -bs additions to tag names.

Update version string and anything else that needs correction.

Describe the bug
When logging into an org the tool can fail without any response in the interface.

To Reproduce
Steps to reproduce the behavior:

1. Attempt to log into a Salesforce org that requires a token and leave it off.
2. Nothing happens

Expected behavior
An error to appear. Something to tell you it failed.

Desktop (please complete the following information):

• OS: Windows 10
• Target Database: n/a

Summary
displayObjectList has a large block of duplication code, plus other features that should be moved to main.

Details
displayObjectList in render.js has a pair of loops over the Org's object list to sort likely selected objects to the top of the list. They should be refactored to at least contain a helper function to reduce duplicate code.

The project either needs a mock that supports the needed elements of electron to allow for testing of dialogs, or needs a library to provide Jest support for electron other ways. This issue should use the same testing approach as #65.

Existing tests of buildSchema send two objects each with two fields. There are only two types of field covered, and none of the preferences options are tested. Please add to the imported schema file a few more fields on each object using additional Salesforce field types (you can get a list of supported field types in app/sf_calls.js), and augment the tests to validate those new fields. Contributions that include coverage of additional details of that feature are encouraged.

Summary
Move SF Org type detection to main thread and handle when describe calls come back.

Details
In render.js there is a snifOrgType function that determines what kind of Salesforce Org you have to help select the right set of default objects. This really should be in the processing context not the UI. Details about what org type it is and recommended default object selection can be added to the response. Those are currently in selectStandardObjects which should also be relocated.

Summary
Since we're pulling and displaying data from a remote source ideally Electron's Sandbox option would be enabled on the main window. The current protections, and the source of the remote data being a fairly secure source itself, mean there is low risk having this disabled. IPC setup changes may be required to make this work smoothly.

Summary
Adding settings to change some behaviors and assumptions about how to translate Salesforce fields.

Please describe the problem you are trying to solve.
Depending on the use case some standard SF fields might be useful in a couple different forms. Having settings to controls those could be helpful. For example, the main record ID field might be more useful in some settings as an auto number, or plain integer than as an SFID. Another example would be RecordTypeIDs being set as the record type name on the assumption it would be translated during later processing.

Proposed Solution

Add the following settings to the preferences:

• Allow Picklists to be Varchar(255) instead of Enums.
• Make Picklist to string when the Salesforce value is unrestricted.
• Include a blank option in all Enums (instead of just null).
• Default lookups fields to indexed.
• Default to make picklists indexed.
• Allow lookups to default to: char(18), varchar(255) (or text when varchars are too numerous).
• Allow setting of default value to field default.
• Default all text values to empty string instead of null.

Additionally:

• #34
• #57
• #58

PRs for any individual piece happily considered. Please do not comment that prs that fix a piece will fix or close this issue.

I’d like to be able to save the output of a schema.
@acrosman what other detail would be helpful to build this out?

Summary
It would be great to minimize to jquery use in the interface to make the code more consistent.

Proposed Solution
Only the json data viewer requires direct use of jQuery. All other instances of $() should be replaced with native JS.

The buttons and the tabs in the rebuilt interface are too similar and are likely to cause confusion. The buttons in particular need to be more button-like to separate them from the tabs clearly.

Right now it uses the default electron Icon, a proper custom icon is needed.

Summary
Add an event subscriber on console messages to capture messages within the primary interface.

Solution
Electron's webContents emits an event when messages are sent the browser console. Since we have a logging console for system messages we should try to subscribe to those events and log those messages in the UI.

Summary
Being able to search the long lists of objects of a specific value is quite useful.

Please describe the problem you are trying to solve.
While the tool does a reasonable job of guessing what objects you want selected, it's useful to be able to quickly jump to the right place to select additional objects or removed selections. So a find in page would be helpful.

Proposed Solution
Something like the approach here or here.