The Data Mining Forum                             open-source data mining software data science journal data mining conferences machine learning in software engineering MLISE 2021 utility mining workshop at ICDM 2021
This forum is about data mining, data science and big data: algorithms, source code, datasets, implementations, optimizations, etc. You are welcome to post call for papers, data mining job ads, link to source code of data mining algorithms or anything else related to data mining. The forum is hosted by P. Fournier-Viger. No registration is required to use this forum!.  
FPGrowth _ Returned non-zero exit status 1
Posted by: thuy4tbn99
Date: January 31, 2021 08:38PM

I use this code:
java -jar spmf.jar run FPGrowth_association_rules transaction.txt output.txt 0.1% 60%

This return error: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f285c000000, 156430761984, 0) failed; error='Not enough space' (errno=12)
My data have 23,000 transactions

Options: ReplyQuote
Re: FPGrowth _ Returned non-zero exit status 1
Date: January 31, 2021 08:45PM

Hi!

Thanks for using SPMF. This means that the algorithm is running out of memory. To avoid this problem you may try increasing the "minsup" parameter of FPGrowth, or using the maximum pattern length constraint (see the documentation). This will reduce the number of possibilities and the algorithm will run faster, use less memory and find less patterns.

The problem is that in itemset mining, the search space exponentially increases usually when you decrease the "minsup" threshold. Thus, I recommend to start with a high value and to see if it runs, and then to decrease it until you find enough patterns.

20k transactions is not a lot. But if the minsup threshold is set too low, the search space can be huge!

If you use the maximum pattern length constraint, it will also reduce the search space a lot and help you to get results more quickly and reduce memory usage!

Best regards

Options: ReplyQuote
Re: FPGrowth _ Returned non-zero exit status 1
Date: January 31, 2021 08:46PM

Actually, I see that you are using the FPGROWTH version for association rule mining...

So you can also try to increase the minconf parameter, and set the constraints on the maximum size of antecedent and consequent of rules! This will help.

Options: ReplyQuote
Re: FPGrowth _ Returned non-zero exit status 1
Posted by: thuy4tbn99
Date: January 31, 2021 10:58PM

Thanks for your reply.

I tried with larger minsup such as 0.008, 0.1 it run oke. But I need minsup is small. I tried with more maximum memory but that still not work.
I don't understand your comment: "set the constraints on the maximum size of antecedent and consequent of rules!"

And can't find any parameter to set it in spmf's documentation. Or spmf has any algorithm for this issue?

Options: ReplyQuote
Re: FPGrowth _ Returned non-zero exit status 1
Date: February 01, 2021 12:29AM

Hi again,

Yes, that features is not mentioned in the documentation. I forgot to add it. I will do later.

There are two optional parameters:
- the maximum antecedent size (e.g 2 items)
- the maximum consequent size (e.g 3 items)

You may add these parameters at the end of the command line like this:

java -jar spmf.jar run FPGrowth_association_rules transaction.txt output.txt 0.1% 60% 2 3

Antecedent means the left part of a rule
Consequent means the right part of a rule

Hope this helps! I thought it was in the documentation but I somehow did not update it. I will update it later.

Options: ReplyQuote
Re: FPGrowth _ Returned non-zero exit status 1
Posted by: thuy4tbn99
Date: February 01, 2021 10:08PM

Thanks for your help,

It seems thats worked well.

Options: ReplyQuote
Re: FPGrowth _ Returned non-zero exit status 1
Date: February 01, 2021 10:25PM

Great!

Options: ReplyQuote


This forum is powered by Phorum and provided by P. Fournier-Viger (© 2012).
Terms of use.