New line character and carriage return in string fields about cobol-to-hive

[datawarlock]

@rbheemana - that is a great idea on the adding inputs for what you wanted to convert the characters to. When I was running my processes I came across the newline and carriage return with the issue. I believe it may also occur for some other characters as well. It may be a good idea to have the process in some sort of list of characters that need to be converted so it is easy to add to in the future.

One other thing I was thinking about was if we add the ability to choose the character we are replacing with (in my case I used space) it would probably be good if it was something set as a table property so that it could be different if you had multiple tables with different requirements. Just a thought on that.

If there is anyway that I can help with any of this let me know. This serde has been a HUGE help for me and I would like to return the favor any way that i can.

[tejasbharambe]

As of now, what I know is that there is nothing like '/n' or newline character in ebcdic file. So how to handle that type of conversion at run time.
Please help me.

[tejasbharambe]

@rbheemana. Kindly help me

[rbheemana]

Please be specific, do you want deserialized hive table to have carriage returns? Or u want to know what happens when carriage return is present in the ebcdic file? Like '\n' present in the file. Are you facing any issue? If yes let me know source input and output.

[tejasbharambe]

Hello, First of all thank you for the reply. Actually we are designing a SerDe for ebcdic to ascii conversion. This is our project for final year.  We haven't started with the execution. Right now we are in the research phase. And what our mentor told is "in ebcdic file format suppose for Eg. Ram123text456 now. Suppose inin our cobol layout we have defined something has name(4) characters and number (3) integer. So in ebcdic file there is no \n which will tell me ohk after ram123 new line is beginning. So how to handle that issue.
I m not sure about this. 

[rbheemana]

So mainframe files are 2 types. Fixed length and variable length. So for fixed length we will length value to determine end of line, for vb file there will be length value stored in the beginning of each record in 4 bytes called RDW. We read this byte and determine length and find end of line.

Hope this helps.

[tejasbharambe]

Yes it did help. Thank you so much.
I will get back to you once I start with the execution.

[tejasbharambe]

I just downloaded all the files from your website. But I have no clue how to run them? Like I have installed Cloudera. But now what? Please guide me. I m new to this. And I m clueless what to do. From where to begin for the project. Basically our Project is to convert the ebcdic files into ascii format at run time. When hive query is fired. SerDe should be activated and run time conversion should occur. Please guide us On 30 Oct 2016 07:34, "tejas bharambe" <[email protected]> wrote: Yes it did help. Thank you so much. I will get back to you once I start with the execution.

[rbheemana]

I believe this is no longer an issue. Please reopen the issue if anyone has data related to it.