It's all about the answers!

Ask a question

[closed] load data to Hive


Hossein Vatani (11) | asked Jan 01 '22, 2:32 a.m.
closed Jan 02 '22, 2:44 p.m. by Ralph Schoon (63.1k33645)

 Hi every one.

I'm trying to load data from dataset into hive with hive-connector but I faced strange behavior. let me explain:
the main hive-connector setting: Record count=2000, Batch size=2000
for dataset with 8 columns and near 1000 rows, it works.
for dataset with 200 columns and 20 million rows, it has strange behavior:
for 10 columns work as charm.
for more than 10 columns, it failed on multiple of Batch size properties with 'IIS-CONN-DAAPI-00099 Hive_Connector_7,0: java.lang.StringIndexOutOfBoundsException: String index out of range: 0 at java.lang.String.substring (String.java: 2667)'
I'm sure this error is not related to String, because for 'Batch size=2000'  the job load near 2000 rows into Hive table and if I increase it to 4000, it loads near 4000 records in hive table.
does any one know the reason of this error?
many thanks

The question has been closed for the following reason: "Question is off-topic or not relevant" by rschoon Jan 02 '22, 2:44 p.m.

One answer



permanent link
Ralph Schoon (63.1k33645) | answered Jan 02 '22, 2:43 p.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

How is this connected to the products ( https://jazz.net/products ) discussed in this forum? I will close this post. If you can provide some background why this is relevant, I would reopen it.


Comments
Hossein Vatani commented Jan 03 '22, 12:33 a.m.

hi, could you please guide me in which forum, could I ask? thanks


Ralph Schoon commented Jan 03 '22, 1:51 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

No, I can't. There is not even enough information for that. If you refer to Apache Hive, try to google if they have a forum or try stackoverflow forums. 


Hossein Vatani commented Jan 03 '22, 2:15 a.m.

it is IBM infosphere problem not Hive issue.I have problem in datastage 


Ralph Schoon commented Jan 04 '22, 2:38 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER