Comments (25)
@djalova Can you please look at it? Thanks
from stocator.
@gilv I have objects with that structure but if I query for "container/data/year=*/month=*"
, it returns the correct results. For which query does this fail?
from stocator.
@djalova thanks. But there is some bug with returning wrong results when there are partitions /year=aaa/month=bbb
and so on. I don't know yet what query return wrong result...can you experiment a bit with it? If i understand correctly then container/data/year=*/month=*
this one work fine for you...may be you can try other options? like to list container/data
or container/data/year=*
?
from stocator.
@gilv For container/data
there needs to be a 0 byte file named container/data
for it to work. The container/data/year=*
should work as is.
from stocator.
I am also seeing an inconsistency with the default file system when there are no results returned from calling ObjectStoreFileSystem.globStatus. An empty array is expected but null is returned instead. I am seeing NullPointerException in many places because of this difference. Can this be changed to return an empty array when there are no results.
from stocator.
@robinnt This should be fixed in master, are you still running into NPEs with the latest code?
from stocator.
@djalova I tried the code in the master branch and the NPEs are fixed. In the hive branch the fixes are not there but there are other fixes that are in progress that are needed. Can you add the NPE fix to the hive branch.
from stocator.
@robinnt The commit that fixes this should be in the hive branch as well. Can you send me a stack trace?
from stocator.
@djalova ok got it. Thanks.
from stocator.
Using the latest code in hive-support branch I am still seeing a problem where a call to globStatus returns an empty array when objects exist in swift that match.
These objects exist in swift:
gendata_19yrs_stoc
gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=2015/month=11
gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=2015/month=11/part-r-00003-attempt_1481248735958_0003_r_000003_0
gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=2015/month=12
gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=2015/month=12/part-r-00004-attempt_1481248735958_0003_r_000004_0
But nothing is returned by
globStatus("/gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=*/month=*")
from stocator.
@robinnt Does it still pick up everything if you query for "gendata_19yrs_stoc*"
?
from stocator.
@djalova Also no results returned with
globStatus("/gendata_19yrs_stoc*")
or
globStatus("/gendata_19yrs_stoc/*")
from stocator.
@robinnt Do you have any logs I can look at? We have tests in TestSwiftOperations.java that mimic your scenario and those are passing. Maybe there is another issue here.
from stocator.
Here is part of the log from calling globStatus until 404 Not Found is returned
2016-12-12 13:44:26,157 DEBUG [CommitterEvent Processor #1] com.ibm.stocator.fs.ObjectStoreFileSystem: Glob status: /gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=*/month=*
2016-12-12 13:44:26,159 DEBUG [CommitterEvent Processor #1] com.ibm.stocator.fs.common.ObjectStoreGlobber: Welcome to glob : /gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=*/month=*
2016-12-12 13:44:26,161 DEBUG [CommitterEvent Processor #1] com.ibm.stocator.fs.ObjectStoreFileSystem: list status: swift2d://stoctables.rn-softlayer/gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=, prefix based true
2016-12-12 13:44:26,161 DEBUG [CommitterEvent Processor #1] com.ibm.stocator.fs.ObjectStoreFileSystem: get file status: swift2d://stoctables.rn-softlayer/gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=
2016-12-12 13:44:26,170 DEBUG [CommitterEvent Processor #1] org.javaswift.joss.command.impl.core.AbstractCommand: JOSS / Sending HTTP HEAD call https://dal.objectstorage.open.softlayer.com/v1/AUTH_b8233702081a4b98ab1cf044db6298bd/stoctables/gendata_19yrs_stoc%2F_DYN0.127403861618094193923682792174938877438%2Fyear%3D
2016-12-12 13:44:26,170 DEBUG [CommitterEvent Processor #1] org.javaswift.joss.command.impl.core.AbstractCommand: * X-Auth-Token=gAAAAABYTxo5E99AydN46thBkZzGcGy0i1lnF8Y5nTN-C4E8gWFBzh5h1T8ruLSleQqt6iXxDGW9RlgDAPP3qg-ZE6lxLK4c-jqlJtnbLOXiWdHPycO2a9hC_B9joILQgZkCngRa8tIvp3JRWsN60mpxNC_93yDSNgy2y0bWc2PdtHRaYcPtIKA
2016-12-12 13:44:26,170 DEBUG [CommitterEvent Processor #1] org.javaswift.joss.command.impl.core.AbstractCommand: * CONNECTION=Keep-Alive
2016-12-12 13:44:26,171 DEBUG [CommitterEvent Processor #1] org.apache.http.client.protocol.RequestAddCookies: CookieSpec selected: default
2016-12-12 13:44:26,171 DEBUG [CommitterEvent Processor #1] org.apache.http.client.protocol.RequestAuthCache: Auth cache not set in the context
2016-12-12 13:44:26,171 DEBUG [CommitterEvent Processor #1] org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Connection request: [route: {s}->https://dal.objectstorage.open.softlayer.com:443][total kept alive: 1; route allocated: 1 of 25; total allocated: 1 of 50]
2016-12-12 13:44:26,171 DEBUG [CommitterEvent Processor #1] org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {s}->https://dal.objectstorage.open.softlayer.com:443][total kept alive: 0; route allocated: 1 of 25; total allocated: 1 of 50]
2016-12-12 13:44:26,171 DEBUG [CommitterEvent Processor #1] org.apache.http.impl.conn.DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 5000
2016-12-12 13:44:26,171 DEBUG [CommitterEvent Processor #1] org.apache.http.impl.execchain.MainClientExec: Executing request HEAD /v1/AUTH_b8233702081a4b98ab1cf044db6298bd/stoctables/gendata_19yrs_stoc%2F_DYN0.127403861618094193923682792174938877438%2Fyear%3D HTTP/1.1
2016-12-12 13:44:26,171 DEBUG [CommitterEvent Processor #1] org.apache.http.impl.execchain.MainClientExec: Target auth state: UNCHALLENGED
2016-12-12 13:44:26,171 DEBUG [CommitterEvent Processor #1] org.apache.http.impl.execchain.MainClientExec: Proxy auth state: UNCHALLENGED
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.headers: http-outgoing-0 >> HEAD /v1/AUTH_b8233702081a4b98ab1cf044db6298bd/stoctables/gendata_19yrs_stoc%2F_DYN0.127403861618094193923682792174938877438%2Fyear%3D HTTP/1.1
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.headers: http-outgoing-0 >> X-Auth-Token: gAAAAABYTxo5E99AydN46thBkZzGcGy0i1lnF8Y5nTN-C4E8gWFBzh5h1T8ruLSleQqt6iXxDGW9RlgDAPP3qg-ZE6lxLK4c-jqlJtnbLOXiWdHPycO2a9hC_B9joILQgZkCngRa8tIvp3JRWsN60mpxNC_93yDSNgy2y0bWc2PdtHRaYcPtIKA
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.headers: http-outgoing-0 >> CONNECTION: Keep-Alive
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.headers: http-outgoing-0 >> Host: dal.objectstorage.open.softlayer.com
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.headers: http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_77)
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.headers: http-outgoing-0 >> Accept-Encoding: gzip,deflate
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.wire: http-outgoing-0 >> "HEAD /v1/AUTH_b8233702081a4b98ab1cf044db6298bd/stoctables/gendata_19yrs_stoc%2F_DYN0.127403861618094193923682792174938877438%2Fyear%3D HTTP/1.1[\r][\n]"
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.wire: http-outgoing-0 >> "X-Auth-Token: gAAAAABYTxo5E99AydN46thBkZzGcGy0i1lnF8Y5nTN-C4E8gWFBzh5h1T8ruLSleQqt6iXxDGW9RlgDAPP3qg-ZE6lxLK4c-jqlJtnbLOXiWdHPycO2a9hC_B9joILQgZkCngRa8tIvp3JRWsN60mpxNC_93yDSNgy2y0bWc2PdtHRaYcPtIKA[\r][\n]"
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.wire: http-outgoing-0 >> "CONNECTION: Keep-Alive[\r][\n]"
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.wire: http-outgoing-0 >> "Host: dal.objectstorage.open.softlayer.com[\r][\n]"
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.wire: http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_77)[\r][\n]"
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.wire: http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2016-12-12 13:44:26,172 DEBUG [CommitterEvent Processor #1] org.apache.http.wire: http-outgoing-0 >> "[\r][\n]"
2016-12-12 13:44:26,303 DEBUG [CommitterEvent Processor #1] org.apache.http.wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
from stocator.
container_e09_1481248735958_0003_01_000001.txt
from stocator.
@robinnt Does your query fail? I didn't see any exceptions thrown. The 404 error occurs because the ObjectStorefileSystem.listStatus() tries to look for objects that match the path exactly, and then when it fails it then decides to look for objects with that prefix. I can fix this behavior, but if you are running into any errors this isn't likely to be the cause. In the logs, the listing eventually returns 8 objects.
from stocator.
That is correct there are no longer any exceptions thrown but the data is not committed into the correct location for the table so the table appears to be empty. This log is from a Big SQL LOAD command where the data is written to a temp directory (gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438) until all map tasks are completed successfully and then the data is moved to the final location (gendata_19yrs_stoc/). The globStatus is called during the move from temp dir to final dir but it incorrectly returns 0 objects so the commit fails with no data moved to the final dir. When the table is queried it is empty because all the data is still under the temp dir and was not properly moved to the table's location.
from stocator.
It seems in the logs there is a delete operation before the listing that causes an exception. Do you know where the delete is coming from? I think it deletes the objects that the next operation looks for.
2016-12-12 13:44:27,256 DEBUG [CommitterEvent Processor #1] com.ibm.stocator.fs.ObjectStoreFileSystem: delete: swift2d://stoctables.rn-softlayer/gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=__HIVE_DEFAULT_PARTITION__/month=__HIVE_DEFAULT_PARTITION__/_logs recursive true. modifed name stoctables/gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=__HIVE_DEFAULT_PARTITION__/month=__HIVE_DEFAULT_PARTITION__/_logs, hostname swift2d://stoctables.rn-softlayer/
...
2016-12-12 13:44:27,339 DEBUG [CommitterEvent Processor #1] com.ibm.stocator.fs.swift.SwiftAPIClient: Listing of stoctables/gendata_19yrs_stoc/_DYN0.127403861618094193923682792174938877438/year=__HIVE_DEFAULT_PARTITION__/month=__HIVE_DEFAULT_PARTITION__/_logs completed with 0 results
2016-12-12 13:44:27,460 DEBUG [CommitterEvent Processor #1] org.apache.hadoop.conf.Configuration: Handling deprecation for all properties in config...
from stocator.
Yes after the globStatus returns an empty list the job tries to clean up and delete a _logs dir. This is expected. After the job ends the objects under the temp dir are still in swift so they were not deleted.
Seems when globStatus is called there should have been some messages with "Candidate accepted" showing the objects found but those messages are missing so something went wrong in the ObjectStoreGlobber.listStatus call where nothing is returned.
from stocator.
Are these files marked if the write operation is successful (i.e. does gendata.../_SUCCESS exist)?
from stocator.
The _SUCCESS file is not created by LOAD
from stocator.
@gilv I think this is caused by this line https://github.com/SparkTC/stocator/blob/master/src/main/java/com/ibm/stocator/fs/swift/SwiftAPIClient.java#L502
From the logs, I can see that when glob() calls APIClient.list() the PaginationMap returns 4 objects, but none of them are returned back to glob. Since there isn't a _SUCCESS file the results in PaginationMap are ignored.
from stocator.
I reran the rest with log level set to ALL so the trace messages are shown. You can see it passes through the code noted above.
container_e09_1481248735958_0007_01_000001.txt
above.
from stocator.
The following lines show that APIClient.list() skips over the objects because _SUCCESS is missing.
2016-12-15 11:03:43,999 TRACE [CommitterEvent Processor #1] com.ibm.stocator.fs.swift.SwiftAPIClient: gendata_19yrs_stoc2/_DYN0.2581013875130941702537108163988541555423/year=2015/month=11 created by failed Spark job. Skipped
...
2016-12-15 11:03:44,209 TRACE [CommitterEvent Processor #1] com.ibm.stocator.fs.swift.SwiftAPIClient: gendata_19yrs_stoc2/_DYN0.2581013875130941702537108163988541555423/year=2015/month=12 created by failed Spark job. Skipped
We currently use this file to differentiate between complete and incomplete files. If it interferes with other operations we might need to find another way to do this. @gilv What do you suggest?
from stocator.
Issue resolved with the introduction of the new object store globber
from stocator.
Related Issues (20)
- Stocator fails with Spark Streaming HOT 1
- mkdir is failing with the following error HOT 20
- FileNotFoundException while accessing objects from hive-style partitioned data stored on an OCS cluster HOT 2
- Upgrade guava from 27.0-jre to 29.0-jre #277
- Upgrade packages that have security vulnerabilities. HOT 4
- Support for Spark 3.x? HOT 5
- Catalog table doesn't work with HIVE-style partitioning if there is no empty object ending with '/' HOT 2
- Using bucket endpoint having virtual-host pattern HOT 1
- HTTP Client Version Warning HOT 1
- Pandas load files written in COS with stocator
- Upgrade hadoop-version from 2.7.3 to 3.2.3
- Upgrade version of jackson_databind and commons-codec as per twistlock scan HOT 2
- Upgrade Amazon SDK HOT 5
- Upgrade version of jackson_databind as per twistlock scan HOT 1
- appstatus file rename failed for spark rolling logs HOT 5
- Spark History Server unable to read rolling logs from COS using stocator
- Double slash in COS parquet path is being added HOT 1
- build failing with java classes not found HOT 6
- support reading credentials from kubernetes secrets
- Get Bucket not found HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stocator.