Recent questions tagged hadoop

0 votes

955 views

1 answer

hadoop - How to access s3a:// files from Apache Spark?

Hadoop 2.6 doesn't support s3a out of the box, so I've tried a series of solutions and fixes, ... Has anyone successfully make both work? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

952 views

1 answer

hadoop - How to Access Hive via Python?

https://cwiki.apache.org/confluence/display/Hive/HiveClient#HiveClient-Python appears to be outdated. When I add this ... Python to access Hive? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

959 views

1 answer

hadoop - Python read file as stream from HDFS

Here is my problem: I have a file in HDFS which can potentially be huge (=not enough to fit all in memory) ... open. Thanks for any help ! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

991 views

1 answer

hadoop - How to fix corrupt HDFS FIles

How does someone fix a HDFS that's corrupt? I looked on the Apache/Hadoop website and it said its fsck command, ... trying to figure it out now. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

991 views

1 answer

hadoop - how many mappers and reduces will get created for a partitoned table in hive

I am always confused on how many mappers and reduces will get created for a particular task in hive. e.g If ... be run during loading the data? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

868 views

1 answer

hadoop - How to use Sqoop in Java Program?

I know how to use sqoop through command line. But dont know how to call sqoop command using java programs . Can anyone give some code view? Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

hadoop - Name node is in safe mode. Not able to leave

root# bin/hadoop fs -mkdir t mkdir: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /user ... -safe-mode-how-to-leave.html?m=1 Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

hadoop - How to open/stream .zip files through Spark?

I have zip files that I would like to open 'through' Spark. I can open .gzip file no problem ... to the CompressionCodecFactory, but am unsuccessful so far. Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

940 views

1 answer

hadoop - How to transpose/pivot data in hive?

I know there's no direct way to transpose data in hive. I followed this question: Is there a way to transpose ... help on how to proceed will be great! Thanks. Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

840 views

1 answer

hadoop - Calling a mapreduce job from a simple java program

I have been trying to call a mapreduce job from a simple java program in the same package.. I tried to refer the ... [])arg.toArray(new String[0])); } } Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

859 views

1 answer

hadoop - Difference between Hive internal tables and external tables?

Can anyone tell me the difference between Hive's external table and internal tables. I know the difference comes ... explain me in terms of nodes please. Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

844 views

1 answer

hadoop - What is the purpose of shuffling and sorting phase in the reducer in Map Reduce Programming?

In Map Reduce programming the reduce phase has shuffling, sorting and reduce as its sub-parts. Sorting is a ... phase in the reducer in Map Reduce Programming? Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

833 views

1 answer

hadoop - Setting the number of map tasks and reduce tasks

I am currently running a job I fixed the number of map task to 20 but and getting a higher ... JobClient: Reduce input records=40000000 [hcrc1425n30]s0907855: Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

816 views

1 answer

hadoop - What is the difference between partitioning and bucketing a table in Hive ?

I know both is performed on a column in the table but how is each operation different. Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

913 views

1 answer

hadoop - Container is running beyond memory limits

In Hadoop v1, I have assigned each 7 mapper and reducer slot with size of 1GB, my mappers & reducers runs fine ... t be assigned more splits than it can handle? Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

850 views

1 answer

hadoop - Write a file in hdfs with Java

I want to create a file in HDFS and write data in that. I used this code: Configuration config = new ... need any permission to write in HDFS? Thanks. Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

840 views

1 answer

hadoop - merge output files after reduce phase

In mapreduce each reduce task write its output to a file named part-r-nnnnn where nnnnn is a partition ID associated ... /reduce merge these files? If yes, how? Question&Answers:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

hadoop - HiveServer2 process failed connection alert

The below hiveserver2 connection failed Alert appears continuously on Ambari server when there is much load ... .com/questions/65865288/hiveserver2-process-failed-connection-alert...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

785 views

1 answer

hadoop - Geomesa-accumulo add index fail job

have a problem with geomesa failed on adding indexes, maybe someones know where problem is? geomesa-accumulo add ... .com/questions/65885199/geomesa-accumulo-add-index-fail-job...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

hadoop - Python Hive Metastore partition timeout

We have ETL jobs in Python (Luigi). They all connect to Hive Metastore to get partitions info. ... //stackoverflow.com/questions/65898409/python-hive-metastore-partition-timeout...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

hadoop - Python connecting to spark in standalone mode and accessing s3 with S3AFileSystem issues

Setup Spark spark-2.4.5-bin-without-hadoop configured with hadoop-2.8.5. Spark is running in standalone ... -to-spark-in-standalone-mode-and-accessing-s3-with-s3afilesyste...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

hadoop - how to remove ADD jar statement in the start of beeline

How can I remove statement that happened when beeline terminal start? I have AD jar statement by default when I start ... /how-to-remove-add-jar-statement-in-the-start-of-beeline...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

736 views

1 answer

hadoop - exception ".internal.jaxp.datatype.XMLGregorianCalendarImpl" during hive query

i get this exception when i perform a large query of hive that does not include any date ... /65913594/exception-internal-jaxp-datatype-xmlgregoriancalendarimpl-during-hive-query...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

990 views

1 answer

hadoop - How to count number of files groupped by date in hdfs

How could we count files from a hdfs folder, groupped by date? Example: 2020-01-03 10:00:00 - file 1 2020- ... /65923492/how-to-count-number-of-files-groupped-by-date-in-hdfs...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.5k views

1 answer

hadoop - Apache Knox Ldap Integration failed

I am using Apacheknox version 1.0.0. I have tried to authenticate knox uiusing Ldap user. I have ... ://stackoverflow.com/questions/65949164/apache-knox-ldap-integration-failed...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

954 views

1 answer

hadoop - Place a subquery in the column list (Hive)

My DB - Hadoop, Hive I have tried to run a query with subquery on the select and it did not run. ... stackoverflow.com/questions/65935897/place-a-subquery-in-the-column-list-hive...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

815 views

1 answer

hadoop - Google Dataproc writing from a client app uses cluster's internal IP for Datanodes

I have setup a Dataproc cluster on Google Cloud. It is sup and running and I can access HDFS and copy files ... -writing-from-a-client-app-uses-clusters-internal-ip-for-datanod...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

858 views

1 answer

hadoop - How often are blocks on HDFS replicated?

I have a question regarding hadoop hdfs blocks replication. Suppose a block is written on a datanode and the DFS ... .com/questions/65946576/how-often-are-blocks-on-hdfs-replicated...

asked Oct 7, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags