The Data Mining Forum                             open-source data mining software data science journal data mining conferences high utility mining workshop
This forum is about data mining, data science and big data: algorithms, source code, datasets, implementations, optimizations, etc. You are welcome to post call for papers, data mining job ads, link to source code of data mining algorithms or anything else related to data mining. The forum is hosted by P. Fournier-Viger. No registration is required to use this forum!.  
How to improve the Spade algorithm for mapreduce
Posted by: Gastel
Date: October 23, 2018 07:26AM

I want design a parallel version of SPADE algorithm for my data. My cloud is not big but I want to use Hadoop or Spark. Please give me advices.

Options: ReplyQuote
Re: How to improve the Spade algorithm for mapreduce
Date: October 26, 2018 06:15AM

Hi, Welcome to the forum

There are different ways for improving SPADE. But since you want to work with map reduce, you could consider a few ideas:
- improve load balancing between cluster nodes
- improve the algorithm or structures
- improve SPADE with new features that are not trivial.

But in any case, there are already quite a few algorithms for mining itemsets in the cloud. You could check them and compare with them.

Regards,

Options: ReplyQuote
Re: How to improve the Spade algorithm for mapreduce
Posted by: Rashid
Date: November 07, 2018 05:01PM

Can I also use map reduce for graph mining?

Options: ReplyQuote
Re: How to improve the Spade algorithm for mapreduce
Date: November 08, 2018 06:41AM

Yes, most likely. The challenge is to develop a parallel version of an algorithm and adapt it for the MapReduce or other big data framework like Spark. You could start from a non parallel algorithm and try to transform it in a parallel algorithm. But not algorithm are easy to transform in a parallel algorithm. Or you could design something new. But there are already some algorithms for big data. So instead of reinventing the wheel you should first check the existing big data algorithms for graph mining.

Options: ReplyQuote


This forum is powered by Phorum and provided by P. Fournier-Viger (© 2012).
Terms of use.