Comments (4)
Results can be observed in the notebook here:
https://github.com/mlbelobraydi/TXRRC_data_harvest/blob/master/Notebooks/TXRRC%20dbf900%2020200901%20working%20with%20bytes.ipynb
from txrrc_data_harvest.
Sorry for the delay responding -- I took a look at the RRC file and it looks like the code is working correctly but that the data they're providing is not actually stored as a negative number. I'm not really familiar with how these coordinate systems work, but is it possible that the RRC is entering it as a positive number and just assuming that we know it should be negative?
That code is checking to see if the last nibble in the binary representation of the number is 0xD; if it is that means that the number is negative (this is how the Signed field is defined in COBOL), and in the database files they provide, it seems like it is usually 0xC.
from txrrc_data_harvest.
but is it possible that the RRC is entering it as a positive number and just assuming that we know it should be negative
@skylerbast, I think that is what is happening. I have added a small if statement that if the signed value isn't negative and the name is longitude to multiply by -1. It isn't ideal, but it is an ok work around. At least the final decimal precision in now being captured. --Matt
from txrrc_data_harvest.
Fixed in the pic_signed function as a work around for TXRRC and closing this issue:
def pic_signed(signed,name,decimal=0): #replacement for pic_latlong and pic_coord
# Converts an EBCDIC Signed number to Python float
# 'signed' must be EBCDIC-encoded raw bytes -- this will not work
# if the data has been converted to ASCII.
## info here http://www.3480-3590-data-conversion.com/article-signed-fields.html
signed_raw = array('B', signed);
val = float(0);
# Bytes 1 to n-1 are stored as plain EBCDIC encoded digits
for i in signed_raw:
val = val * 10 + (i & 0x0F)
# If the penultimate nibble == 0xD, then the number is negative. Otherwise,
# it is either positive or unsigned.
val = (val * (-1 if signed_raw[-1] >> 4 == 0xD else 1)) / 10**decimal
##TXRRC signs longitude as a positive value with 0xC and requires transformation to a negative number
if 'LONGITUDE' in name and val > 0: ##This is only appropriate for western hemisphere
val = -val
return val
from txrrc_data_harvest.
Related Issues (17)
- Identify the codec for the IBM mainframe files that can be read in Python HOT 2
- Change the dbf900_formats section to work with bytes instead of string to keep lat-long accuracy. HOT 2
- Comp-3 function HOT 4
- Preserve original entry of date infomation along with conversion
- Oil Production Layout HOT 3
- bad add: Comp-3 Function HOT 1
- Request for how to setup to start helping with development. This should be in the wiki and have links to things like Anaconda, Github, and basic Python resources. HOT 5
- Gas Production Layout HOT 2
- Testing file to read data from gas ebcdic file to pandas dataframes HOT 2
- Workflow documentation? HOT 1
- TXRRC file location changes HOT 2
- Request: Add support for working with Polars (in addition to Pandas) HOT 2
- Request: Add support for installing via `pip` by publishing to PyPi HOT 1
- Create definition libraries for all 28 sections in the dbf900.ebc.gz file HOT 7
- Create script that tests the definitions for the files and reads the data to something that can be loaded to SQL. HOT 7
- supporting systems with limited memory. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from txrrc_data_harvest.