hadoop

[Solved] Spark in Business Intelligence

by

Introduction Business intelligence (BI) is a critical tool for organizations to gain insights into their data and make informed decisions. Spark is an open-source distributed computing platform that has become increasingly popular for its ability to process large amounts of data quickly and efficiently. Spark is a powerful tool for business intelligence, as it can … Read more

[Solved] What should I use to perform similarity functions on 200 column 12 million row dataset? [closed]

by

After getting suggestions from a couple of friends, I looked up the documentation on ElasticSearch. Seems like that’s the perfect tool for my use-case. It’s built for search/retrieval needs such as this, shards like anything, can handle huge data. Here’s what should be done: Store each row in a document, with the key elements being … Read more

[Solved] Could you please tell me where I would find the output for the mapreduce program Wordmedian in hadoop? Is it stored in a directory in HDFS?

by

Could you please tell me where I would find the output for the mapreduce program Wordmedian in hadoop? Is it stored in a directory in HDFS? solved Could you please tell me where I would find the output for the mapreduce program Wordmedian in hadoop? Is it stored in a directory in HDFS?

[Solved] Process unstructured and multiple line CSV in hadoop

by

Because you’re coping with multi-line data you cannot use a simple TextInputFormat to access your data. Thus you need to use a custom InputFormat for CSV files. Currently there is no built-in way of processing multi-line CSV files in Hadoop (see https://issues.apache.org/jira/browse/MAPREDUCE-2208), but luckily there’s come code on github you can try: https://github.com/mvallebr/CSVInputFormat. As far … Read more

[Solved] MapReduce to Spark

by

This is a very broad question, but the short of it is: Create an RDD of the input data. Call map with your mapper code. Output key-value pairs. Call reduceByKey with your reducer code. Write the resulting RDD to disk. Spark is more flexible than MapReduce: there is a great variety of methods that you … Read more

[Solved] How to get the specified output without combineByKey and aggregateByKey in spark RDD

by

Here is a standard approach. Point to note: you need to be working with an RDD. I think that is the bottleneck. Here you go: val keysWithValuesList = Array(“foo=A”, “foo=A”, “foo=A”, “foo=A”, “foo=B”, “bar=C”,”bar=D”, “bar=D”) val sample=keysWithValuesList.map(_.split(“=”)).map(p=>(p(0),(p(1)))) val sample2 = sc.parallelize(sample.map(x => (x._1, 1))) val sample3 = sample2.reduceByKey(_+_) sample3.collect() val sample4 = sc.parallelize(sample.map(x => (x._1, … Read more

[Solved] Installing Hadoop in LinuxMint

by

can install the VM on linux You can use a VM on any host OS… That’s the point of a VM. The last link is only Hadoop, where Hortonworks has much, much more like Spark, Hive, Hbase, Pig, etc. Things you’d need to additionally install and configure yourself otherwise Which is better for learning and … Read more

[Solved] Can’t copy file into HDFS

by

You should provide specific details like the exception you get, steps you follow etc, Since you have not specified any information at all, i would say check for the config files to make sure you have all the required entries in corresponding files : In core-site.xml you should have <configuration> <property> <name>fs.default.name</name> <value>hdfs://ipaddress:port</value> </property> <property> … Read more