bullhorn / dataloader Goto Github PK

Hello again. I've been inserting a few client corporation and client contact records using DataLoader. So far, I have added 15 client corporation records, all of which are listed in my DataLoader results as successfully inserted. One record, ID 13, is not visible in the Bullhorn web UI. When I try to insert client contact records for that client corporation, I get an error that the client corporation doesn't exist on Bullhorn. I checked that client corp ID 13 exists on Bullhorn by issuing a "query" command via Postman; the query returned the details of client corp 13 OK. However, when I issued a "search" command via Postman, no results were returned. So it looks as though that one record was not indexed for some reason. But that prevents the client contacts being inserted. Are there any reasons why the indexing would have failed on this one record? Is there any to force the DataLoader to use a "query" rather than a "search" to get the client corp?

It might be a good idea down the road to use the official rest sdk

Support Linux, Mac, and windows

Dataloader errors when more than 15 characters are entered for the leadSource field on the Lead entity. Database allows for 100 characters on this field.

Complete domain model

When loading file attachments to the Candidate record I received a generic error (see below) when there was a candidate record deleted and one that was not deleted with the same external id field value. The workaround was to run the index and it resolved this issue. However, this is still a bug. It should not take into account deleted records when doing file attachments.
Here is the error:
com.bullhornsdk.data.exception.RestApiException: {
"errorMessageKey" : "errors.internalError",
"errorMessage" : "An internal error has occurred"
}
I also ran an update to the candidate record and it considered the deleted records as a duplicate. In this case the error made more sense, but still should not occur. Here is the error:
com.bullhornsdk.data.exception.RestApiException: Row 4: Cannot Perform Update - Multiple Records Exist. Found 2 Candidate records with the same ExistField criteria of: {externalID=3850884}

Implement oauth workflow for machine based auth. Right now, existing token is passed in at runtime

Hi, this is not so much an issue, more a request for clarification. What date formats are supported in the CSV data? Also, what text qualifiers are OK to use in the CSV?

Provide a way of reporting descriptive errors of which records failed to the user

In master data service

In master data service

Hello again. Sorry for creating another issue to ask another question. Is there a better forum in which to ask these questions? Anyway, I was wondering about the address.countryID field. I've been using the metadata from the Bullhorn API to generate empty database tables with the same structure as the API. The countryID field in the API is presented as an int, so my blank database has a numeric countryID field. Naturally, I then used the Bullhorn numeric ID for the countries in my CSV for DataLoader. In my example, I had 2306 (Netherlands) and 2202 (Belgium). However, when I loaded my CSV, the country was set to USA. I then read in your documentation that countryID should be set to the name of the country as listed in the Bullhorn data accessed via the API. So I set the countryID to "Netherlands" and "Belgium" and loaded the data via DataLoader and the countries were set correctly. Which is great! However, I was thinking that it would be better to have separate fields for country ID and name, so that it matches the API. Otherwise I will need to include some logic in my program to specifically handle the country stuff differently from the other fields when I'm creating my database structure. Also, I have to query the API to get the Bullhorn-canonical names of all the countries anyway, so getting the ID instead is just as easy as getting the names. Thanks for the ongoing help and apologies for the length this time!

https://github.com/bullhorn/dataloader/blob/master/dataloader/pom.xml

Remove eclipse dependencies

Currently when loading entities with to-many fields, if there are multiple associations to be made these will be sent as multiple individual calls (which takes around half a second each). Can this be changed to a single call for the whole set?

This is most evident for example when uploading a candidate with many skills (e.g. 20+).

There are occasional errors in the results success file for ClientCorporationCustomObjectInstance1 file that result in a missing ID cell. Attempting to delete the results file returns:

'ERROR: java.io.IOException: Header column count 10 is not equal to row column count 9'

and then hangs.

Complete domain model for each entity to include all fields and associations

Im working with 1Staff on a two-way integration. Part of the integration, which sends data back to BH, requires loading data into a defined Custom Object. Their developer has reporting the following issue w/ below stack trace:

**Thank you – that is quite helpful.

I have been able to get the dataloader to work with ClientContacts. (I was able to import about 5K records in 7:24).

However, I have not had any success with the CustomObject1. Am I missing something simple? (I don’t have access to the documentation yet so I apologize if I am.)

Attached is the test import file I have been using (and in the properties file, I have ClientCorporationCustomObjectInstance1ExistField=id) Below is the output I receive:

c:\BH\dataloader-master\dataloader-master>dataloader.bat load C:\BH\dataloader-master\dataloader-master\data\ClientCorporationCustomObjectInstance1.csv
Loading ClientCorporationCustomObjectInstance1 records from: C:\BH\dataloader-master\dataloader-master\data\ClientCorporationCustomObjectInstance1.csv...
Exception in thread "pool-2-thread-1" Results of DataLoader run
java.lang.NullPointerExceptionStart time: 2017-05-09 07:20:30

    at com.bullhorn.dataloader.service.csv.CsvFileWriter.writeRow(CsvFileWriter.java:82)
    at com.bullhorn.dataloader.task.AbstractTask.writeToResultCSV(AbstractTask.java:90)End time: 2017-05-09 07:20:42
    at com.bullhorn.dataloader.task.LoadCustomObjectTask.run(LoadCustomObjectTask.java:67)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)Args: load C:\BH\dataloader-master\dataloader-master\data\ClientCorporationCustomObjectInstance1.csv

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

Total records processed: 0
Total records inserted: 0
Total records updated: 0
Total records deleted: 0
Total records failed: 0
Finished loading ClientCorporationCustomObjectInstance1 records in 00:00:09**

1Staff is working within their Professional AdvantageDEMO corporation and uses the allocated profad.demoapi username.

They currently have two Custom Objects setup. The one mentioned in the stack trace above is CustomObject1 (Invoice History).

Can this be looked into?

If header row doesn't match field name in meta, look to see if it matches user friendly "Label". If neither matches, check against the entity's field mapping.

Deleting of custom objects should also work with whatever exist field has been configured as well.

I had billingAddress.address1,billingAddress.city,billingAddress.state,billingAddress.zip,billingAddress.countryID as part of my csv but the data load didn't add values to these fields.

Hello,

Not sure if this was by design or not but currently all of the ExistFields are saved with the # in front of it (turns it green in notepad++ like it is being commented out) and when running an update based on id and updating a single field it inserts a new record with only the the one field. Tested this two times each way and it inserted both times with the # still on the end.

Removing the # allows it to update just fine. Not sure if this is as designed but being a change from the previous versions I assumed not.

If header row doesn't match field name in meta, look to see if it matches user friendly "Label"

Currently privateLabelId is decided via a properties file, should probably be received via the settings call.

Hello again. I've been trying to load some notes. A value for personReference is required, but I don't know what this should be? In your example, you have personReference.name as the field name and "Manager CorporateUser" as the example value. PersonReference doesn't seem to have an ExternalID field, so I guess it can't be the candidate/contact. The commentingUserID is in a different field and is also required (I've set it to Unassigned). If it's not the candidate and it's not the commenting user, I don't see what else the PersonReference could be...? Thanks again for your help.

Finish - complete domain model

It seems to have started with 2.0.0 but when making an update to ClientContacts it will error out unless you also include category in the update as well (I updated with category.name).

So, for example, in REST (and in 1.4 / 1.0.2 I believe) you can update a ClientContact's status and id alone. To do it in 2.0.0 you need status, id, and category.

Complete and test all to-many associations

If the application execution takes longer than the expiration timeout of the rest token then it will quit working suddenly. We should implement a thread-safe way of handling expiration.

Would it be possible to add the ability to configure the structure of the name of the result file? I ask because the Microsoft CSV ODBC driver cannot read files with hyphens or more than one period in the name. Given that the driver has been like that for the past decade, I doubt it will get fixed soon! So in the meantime, it would be great to be able to have result files without hyphens or periods. Thanks :)

Got the following Java exception when loading an unusually long note:

ERROR: java.io.IOException: Maximum column length of 100,000 exceeded in column 2 in record 0. Set the SafetySwitch property to false if you're expecting column lengths greater than 100,000 characters to avoid this error.

The exception didn't get caught or logged, there was nothing for that note in the success/failure output CSVs. It also stopped any further processing, although the DataLoader program did not exit, just sat there idle.

UI should include:
Import Type
File Picker
Input fields for fields currently in the properties file

Complete domain model

We discussed having TravisCI for build health and building the release artifacts.

I would be great if the data loader would accommodate for tearsheets. It is fairly common for clients to have lists of candidates (and to a lesser extent contacts) within their system as hot lists that they require being brought over. I am wondering what the feasibility of having this added would be?

For example, in v2.1.0 the CSV template example business sectors for candidate are written with header "businessSectors.name" and data type "String". In v2.1.1, the header is "businessSectors.id" with data type "Integer". This seems misleading to me because the content of that field should always be a comma-separated list of names of business sectors, not a list of numeric IDs? Or have I missed something...?

Hello again. I'm having some trouble understanding how to tackle notes. There are two things I'm struggling with.

1. When I load notes, I can't specify an external ID, nor can I get the external ID back in the results. So I have to use migrateGUID. This is really invonvenient when we are dealing with multiple data cuts. Pretty much every migration will have multiple data cuts: some data will change and some will stay the same. It is much more convenient (and safer) to use an ID which is consistent between data cuts and is contained in the data (externalID) rather than generated by our scripts (migrateGUID).
2. The other problem I'm having is that I can't get Update to work on notes. I've tried specifying migrateGUID and ID as the noteExistField, but neither worked, DataLoader always does an insert.

I am open to suggestions for handling the notes in a different way. No doubt you had a particular approach in mind when creating DataLoader...

Hello,

I've setup DataLoader on my machine in order to troubleshoot an issue and I have gotten stuck where it is giving me the following error (the same happens when I am attempting to create new and update existing):

Apr 22, 2016 4:39:16 PM com.bullhorn.dataloader.service.api.BullhornAPI frontLoa
d
INFO: Front loading clientContact
Exception in thread "main" org.json.JSONException: JSONObject["fields"] not foun
d.
        at org.json.JSONObject.get(JSONObject.java:477)
        at org.json.JSONObject.getJSONArray(JSONObject.java:624)
        at com.bullhorn.dataloader.service.api.BullhornAPI.getFieldsFromMetaResp
onse(BullhornAPI.java:369)
        at com.bullhorn.dataloader.service.api.BullhornAPI.getMetaDataTypes(Bull
hornAPI.java:316)
        at com.bullhorn.dataloader.service.api.BullhornAPI.getRootMetaDataTypes(
BullhornAPI.java:486)
        at com.bullhorn.dataloader.Main.loadCsv(Main.java:68)
        at com.bullhorn.dataloader.Main.main(Main.java:41)

I'm not certain what information to give so let me know to add more if needed.

Admin Command Prompt line:

C:\Users\ccoffman\Desktop\dataloader>java -Dpropertyfile=C:\Users\ccoffman\Desktop\dataloader\dataloader.properties -jar dataloader-importer-1.0.1.jar clientContact C:\Users\ccoffman\Desktop\dataloader\export.csv

Properties file (attempting to create new clientContacts):

• I included all non-nullable fields

clientContactExistField=type,categories.id,owner.id,phone,status,lastName,firstName,name,email,clientCorporation.id,preferredContact,isDeleted,numEmployees

frontLoadedEntities=
numThreads=1
dateFormat=MM/dd/yyyy
listDelimiter=|
pageSize=500
cacheSize=10000

username=(user name)
password=(password)

authorizeUrl=https://auth.bullhornstaffing.com/oauth/authorize
tokenUrl=https://auth.bullhornstaffing.com/oauth/token
loginUrl=https://rest.bullhornstaffing.com/rest-services/login

clientId=workingID
clientSecret=workingSecret

Above is the excel file I am loading through DataLoader

The folder that all of these files reside in is C:\Users\ccoffman\Desktop\dataloader

I appreciate any help and feel free to reach out to me via Slack for any discussion as well.

It was also suggested that it could be a field mapping / user type entitlement issue. The user type has full access to API functionality and I am not certain what to check since they run off of the field names not labels.

Thank you,
Colin

Test all data types during import to ensure proper conversion

Occasionally there are unsuccessful runs and if re-run, the process will often complete successfully.

Example Run:

Loading ClientCorporationCustomObjectInstance2 records from: data\ClientCorporationCustomObjectInstance2.csv...
Exception in thread "pool-2-thread-2" java.lang.NullPointerException
Results of DataLoader run at com.bullhorn.dataloader.service.csv.CsvFileWriter.writeRow(CsvFileWriter.java:82)

    at com.bullhorn.dataloader.task.AbstractTask.writeToResultCSV(AbstractTask.java:110)Start time: 2017-05-30 14:42:27

    at com.bullhorn.dataloader.task.LoadCustomObjectTask.run(LoadCustomObjectTask.java:68)End time: 2017-05-30 14:42:30

    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)Args: load data\ClientCorporationCustomObjectInstance2.csv

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)Total records processed: 1

    at java.lang.Thread.run(Unknown Source)Total records inserted: 1

Total records updated: 0
Total records deleted: 0
Total records failed: 0
Finished loading ClientCorporationCustomObjectInstance2 records in 00:00:02

Hard-coded for bhnext now, needs to be externalized

Error below:

INFO: Querying for https://rest0.bullhornstaffing.com/rest-services/yh99g/search /Candidate?fields=id&query=isDeleted%3A%22false%22+AND+name%3A%22%27Lorenzo+Liga to%27%22+AND+id%3A%2216637%22+AND+email%3A%22%27lorenzoligato%40icloud.com%27%22 &count=2&useV2=true&BhRestToken=cad4acbc-147c-479b-9359-f9b062c9e7e1 Feb 25, 2016 8:31:43 AM com.bullhorn.dataloader.service.api.BullhornApiUpdater g etCall INFO: Querying for https://rest0.bullhornstaffing.com/rest-services/yh99g/query/ Candidate?fields=id&where=id%3D16637&count=2&BhRestToken=cad4acbc-147c-479b-9359 -f9b062c9e7e1 Exception in thread "pool-1-thread-3" com.google.common.util.concurrent.Unchecke dExecutionException: org.json.JSONException: JSONObject["count"] not found. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.j ava:4824) at com.bullhorn.dataloader.service.executor.JsonService.createOrGetEntit y(JsonService.java:95) at com.bullhorn.dataloader.service.executor.JsonService.run(JsonService. java:55) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.json.JSONException: JSONObject["count"] not found. at org.json.JSONObject.get(JSONObject.java:477) at org.json.JSONObject.getInt(JSONObject.java:604) at com.bullhorn.dataloader.service.api.BullhornApiUpdater.idExistsInRest (BullhornApiUpdater.java:99) at com.bullhorn.dataloader.service.api.BullhornApiUpdater.idExistsButNot InRest(BullhornApiUpdater.java:94) at com.bullhorn.dataloader.service.api.BullhornApiUpdater.merge(Bullhorn ApiUpdater.java:28) at com.bullhorn.dataloader.service.query.EntityCache.load(EntityCache.ja va:57) at com.bullhorn.dataloader.service.query.EntityCache.load(EntityCache.ja va:37) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(L ocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2 319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache .java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ... 8 more

I just ran a test where I added references to 250 Candidates then searched for them in the system via "with references" and they all returned however their number was 0. After running an indexing event on the reference in the edit screen for the reference it displayed properly in the list with a 1.

I received an issue about after using the dataloader the client was getting timeout errors running rest calls, like checking subscriptions. I verified the call is of the correct format and this is an example error:

"rootCause":{"errorMessage":"Read timed out"},"mostSpecificCause":{"errorMessage":"Read timed out"},"errorMessage":"Could not access HTTP invoker remote service at [http:\/\/bhservice9:8888\/data-event-services\/sr\/DataEventService]; nested exception is java.net.SocketTimeoutException: Read timed out"

I ran the updated dataloader on my test user / corp and can no longer get rest access with that user. When I login manually I get as far as validating my access token then I get the invalid or expired Oauth token.
I generated new credentials for REST and they work for other API users but not my test user.

Any ideas on why or how to revert this?

Thanks

Last time I've heard we decided on GPLv3 or Affero GPL.

Readme should also be updated to tell users where they can locate file in OSX. I had to create the directory structure it was kept spitting out as the missing file location, though it did not already exist and was in hidden areas.

Current version defaults to 'US' when using country codes in ClientContact address.countryID. It would be useful if both code & name is supported.

I realise this is sort of a case of "not holding it properly", but I've been trying to add candidate records with more than 20 business sectors (it's a quirk of the test data I've been working with). When I try to load the candidate, I get an error which is not accurate: "businessSectors does not exist with id of the following values". In fact the business sectors do exist, it just reports this error for each additional business sector over the 20 limit. I was able to verify this with two records, both of which had 20+ business sectors. When I reduced the number to 19, leaving in the business sectors which had previously been reported as not existing, both records inserted OK with all 19 business sectors. Assuming that there is a hard limit in Bullhorn of 20 business sectors (is that the case?), would if be possible for DataLoader to report that the limit is being exceeded rather than the error about business sectors not existing?

Even when candidate.externalID is used the personreference is required (either ID or Name)
Here is the erro

C:\Users\joel_\Downloads\dataloader\dataloader>dataloader load data\note.csv
ERROR: java.lang.IllegalArgumentException: DataLoader Properties Error: numThreads property must be in the range of 1 to 10

C:\Users\joel_\Downloads\dataloader\dataloader>dataloader load data\note.csv
Loading Note records from: data\note.csv...
ERROR: com.bullhornsdk.data.exception.RestApiException: {
"message" : "error persisting an entity of type: Note",
"errorMessageKey" : "errors.cannotPersistEntity",
"errorCode" : 500,
"errors" : [ {
"detailMessage" : "missing required property: personReference",
"propertyName" : "personReference",
"severity" : "ERROR",
"type" : "MISSING_REQUIRED_PROPERTY"
} ],
"entityName" : "Note",
"errorMessage" : "error persisting an entity of type: Note"
}

Provide a way to create a template csv for each entity