Apache Hadoop Puns

why use Hadoop MapReduce transformation if you can use Apache Spark and call map transformation and then call reduce?

I'm working on a textbook practice project for a class where I need to find the maximum hourly electricity consumption and the average daily electricity consumption. Here is some sample data (the ENERGY_READING column is cumulative), these are records inside of a text file:

Here is what I have for the maximum hourly consumption, but I'm not sure how to take into account that the energy levels are cumulative, and I need to somehow also display the hour in the results.

String filepath = "/Sparkresults/MaximumConsumption";

SparkConf conf = new SparkConf().setMaster(some ip address).setAppName("Maximum         
 Hourly Consumption");

JavaSparkContext sc = new JavaSparkContext(conf);

JavaRDD<String> textFile = sc.textFile("/data/energydata");

// Remove header row
String header = textFile.first();
textFile = textFile.filter(row -> !Objects.equals(row, header));


JavaPairRDD<String, Double> results =  textFile.map(ln -> ln.split("\t"))
        .mapToPair(rec -> new Tuple2<>(rec[1], Double.parseDouble(rec[4])))
        .reduceByKey(Math::max);


results.saveAsTextFile(filepath);

And I have no idea how to even begin the average.

Any help would be much appreciated.

I prefer to learn via reading, so looking forward to text based learning resources. Something like mozilla guide for web developer. Please help .

https://www.infoworld.com/article/3615695/it-s-pythons-all-the-way-down.amp.html says

"Apache products, like Hadoop, Hive, and Spark, continue to decline in importance."

If that's true, why and what is replacing spark?

Thanks.

I often hear about these technologies being mentioned in the datascience industry. How important are these to know vs knowledge of statistics/ml algorithms, regular SQL/r/python?

Ok, so I have done a Udemy course on Spark(pyspark) and Databricks, I am reading books on Spark, and have done countless google searches. But I am still confused about what and how these different components work and cannot find a plain answer online.

I really hope someone will be able to help me with even the simplest of answers you could provide on these questions I have. They're really starting to confuse me and they feel like such basic things to know.

What I have read is Apache, is a web server that communicates between browsers and servers, as a middle man.

• What does that mean for Apache Spark , Apache Hadoop and all the other Apache things out there? Does the use of Apache just mean they use the Apache web server?

Then I didn't understand, why does Spark use Hadoops File System if they they considered competitors who do the same thing? Do they both do a similar thing, and have the same goal?

Do you have to use Databricks to use Spark? Does spark always utilise databricks clusters, or can spark be used in a different way such as when using pyspark maybe with another provider through AWS? What else can you use Spark with instead of databricks if its possible?

Is Palantir Foundry the same type of product as Databricks, do they both have the same purpose?

Ok these were my burning questions. Would be so grateful if someone could explain these for me, would be a big help. Thank you!

[Copy of SO question, seeking info from users]

We are implementing a Spark client for direct access to lakeFS. This is a Git-like (versioned) storage layer on top of some other object store. We would like our file system to supply Spark (and other Hadoop-based tools) the ability to handle URLs such as `lakefs://repo/branch/path/to/object`.

Package org.apache.hadoop.fs supplies both AbstractFileSystem and an older FileSystem (links intentionally point at docs for an older version of Hadoop, to include all of our users). Any recommendations for which to implement?

1. Is there any reasonable expectation by Spark users for current code to rely on being able to generate specifically a FileSystem (the older type)? Specifically, do you have (or know of any) code that relies on FileSystem?
2. Is implementation of either of these options significantly more robust (or simple)?
3. Can the same code support both types (presumably with different configuration options)?

Thanks for any pointers!

Hey folks,

I'm currently looking for/researching ways of making on-prem Apache Hadoop/Spark clusters more cost- and resource-efficient. A total noob here, but my findings now go like this:

- you should migrate to the cloud, be it as-is or with re-architecture

- you better migrate to Amazon EMR 'cause it offers low cost, flexibility, scalability, etc.

What are your thoughts on this? Any suggestions?

Also, I'd really appreciate some business (not technical) input on whitepapers, guides, etc. I could read to research the topic, to prove that my findings are legit. So far, I found a few webinars (like this one - https://provectus.com/hadoop-migration-webinar/ ) and some random figures at the Amazon EMR page ( https://aws.amazon.com/emr/ ), but I fear these are not enough.

Anyway, I'd appreciate your thoughts and ideas. Thanks!

Noob question. What is the difference between:

• Apache Spark
• Kubernetes
• Hadoop or Hadoop/Yarn