[closed] load data to Hive
Hossein Vatani (11)
| asked Jan 01 '22, 2:32 a.m.
closed Jan 02 '22, 2:44 p.m. by Ralph Schoon (63.1k●3●36●45) Hi every one.
I'm trying to load data from dataset into hive with hive-connector but I faced strange behavior. let me explain:
the main hive-connector setting: Record count=2000, Batch size=2000
for dataset with 8 columns and near 1000 rows, it works.
for dataset with 200 columns and 20 million rows, it has strange behavior:
for 10 columns work as charm.
for more than 10 columns, it failed on multiple of Batch size properties with 'IIS-CONN-DAAPI-00099 Hive_Connector_7,0: java.lang.StringIndexOutOfBoundsException: String index out of range: 0 at java.lang.String.substring (String.java: 2667)'
I'm sure this error is not related to String, because for 'Batch size=2000' the job load near 2000 rows into Hive table and if I increase it to 4000, it loads near 4000 records in hive table.
does any one know the reason of this error?
many thanks
|
The question has been closed for the following reason: "Question is off-topic or not relevant" by rschoon Jan 02 '22, 2:44 p.m.
One answer
Ralph Schoon (63.1k●3●36●45)
| answered Jan 02 '22, 2:43 p.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER How is this connected to the products ( https://jazz.net/products ) discussed in this forum? I will close this post. If you can provide some background why this is relevant, I would reopen it.
Comments
Hossein Vatani
commented Jan 03 '22, 12:33 a.m.
hi, could you please guide me in which forum, could I ask? thanks No, I can't. There is not even enough information for that. If you refer to Apache Hive, try to google if they have a forum or try stackoverflow forums.
Hossein Vatani
commented Jan 03 '22, 2:15 a.m.
it is IBM infosphere problem not Hive issue.I have problem in datastage Do you know Google? Google for "ibm infosphere forum" finds https://www.ibm.com/mysupport/s/question/0D50z00006HvphH/support-qas-are-moving-many-of-our-infosphere-datastage-forum-questions-and-answers-are-being-migrated-to-a-new-support-forum?language=en_US which provides a link to some kind of support forum. |