This is a forum for anyone interested in Artificial Intelligence and Data Mining! http://forum.ai-directory.com/index.php Thu, 17 May 2012 05:24:12 -0400 Phorum 5.2.10 http://forum.ai-directory.com/read.php?5,572,574#msg-574 http://forum.ai-directory.com/read.php?5,572,574#msg-574
But this is a different problem than sequential pattern mining]]> webmasterphilfv The Data Mining Forum Wed, 16 May 2012 23:18:02 -0400 http://forum.ai-directory.com/read.php?5,572,573#msg-573 http://forum.ai-directory.com/read.php?5,572,573#msg-573
Welcome to the forum.

Sequential pattern mining like GSP take as input : (1) a sequence database and (2) a parameter called "minsup".

A sequence database is a set of sequences.

The goal of sequential pattern mining is to find subsequences that are common to several sequences.

In your case, I think that you have only a single sequence.


I will give you an example of a sequence database.

For example, here are four sequences named s1, s2 and s3:

s1: (a), (b, c), (d), (e)
s2: (b), (d), (e), (f)
s3: (b), (d), (e), (f)

The first sequence means that item "a" occured, and was followed by "b" and "c" at the same time, then followed by "d" and then by "e"

The second sequence means that "b" was followed by "d", followed by "e" followed by "f.

The third...

So if you have a sequence database, you can find some subsequences that are common to several sequences with algorithms like GSP and PrefixSpan.

For example, those algorithms could find that the subsequence (b), (e) appears in s1, s2 and s3.

Another example is that (e), (f) appears in s1 and s2.

So I just write this quickly to give you an idea of what those algorithms do.

If you have only a single sequence these algorithm may not be appropriate unless you can divide your sequence in several sequences and that it make sense for your application to do that.

Best,

Philippe]]> webmasterphilfv The Data Mining Forum Wed, 16 May 2012 23:15:31 -0400 http://forum.ai-directory.com/read.php?5,572,572#msg-572 http://forum.ai-directory.com/read.php?5,572,572#msg-572
I am newbie in the sequential patterns mining. Right now, i am doing my internship and i will implement this method.

I need help for the datasets that will fit for the sequential pattern with time-series.

I have prepared an example of datasets like below :

Time M_T_ambiante DeltaT M_T_Ext M_Rayonnement
20/06/2011 00:00 Medium Normal Medium Low
20/06/2011 00:05 Medium Normal Medium Low
20/06/2011 00:10 Medium Normal Medium Low
20/06/2011 00:15 Medium Normal Medium Low
20/06/2011 00:20 Medium Normal Medium Low
20/06/2011 00:25 Medium Normal Medium Low
20/06/2011 00:30 Medium Normal Medium Low

Is this datasets already can be used with sequential patterns like GSP or else?

Thanks before your help.

Regards,
Yogi]]> YougyZ The Data Mining Forum Wed, 16 May 2012 05:36:54 -0400 http://forum.ai-directory.com/read.php?5,322,571#msg-571 http://forum.ai-directory.com/read.php?5,322,571#msg-571 Lyon, France, on 29-31 October 2012.
Deadline: 25th May 2012
Notification: 8th July, 2012
Format: (up to) 15 pages in Springer LNCS format]]> webmasterphilfv The Data Mining Forum Mon, 14 May 2012 08:59:14 -0400 http://forum.ai-directory.com/read.php?5,33,570#msg-570 http://forum.ai-directory.com/read.php?5,33,570#msg-570
Try run with these for help:

ibm_gen_file.exe seq -help]]> tisonet The Data Mining Forum Sat, 12 May 2012 05:14:30 -0400 http://forum.ai-directory.com/read.php?5,33,569#msg-569 http://forum.ai-directory.com/read.php?5,33,569#msg-569
What kind of error did you get?

Philippe]]> Philippe The Data Mining Forum Fri, 11 May 2012 10:21:47 -0400 http://forum.ai-directory.com/read.php?5,33,568#msg-568 http://forum.ai-directory.com/read.php?5,33,568#msg-568 doris The Data Mining Forum Fri, 11 May 2012 10:05:50 -0400 http://forum.ai-directory.com/read.php?5,322,567#msg-567 http://forum.ai-directory.com/read.php?5,322,567#msg-567 [www.kdir.ic3k.org]
October 4 - 7, 2012, Barcelona, Spain
Format: see webpage]]> webmasterphilfv The Data Mining Forum Wed, 09 May 2012 18:36:51 -0400 http://forum.ai-directory.com/read.php?5,520,566#msg-566 http://forum.ai-directory.com/read.php?5,520,566#msg-566
i got so many things now and clear with it.

yes philippe sir i also feel that algorithm run slow when memory reach at the highest point. like you mention 1 GB]]> Dvijesh88 The Data Mining Forum Wed, 09 May 2012 07:55:01 -0400 http://forum.ai-directory.com/read.php?5,546,565#msg-565 http://forum.ai-directory.com/read.php?5,546,565#msg-565
These are some good topics!

Graph mining are a very good topic in my opinion. It may need a little bit more mathematics than some other topics. But it should be interesting. In particular, social network mining is very popular now.

Multimedia is also very promising. For example, mining video stream from camera to detect abnormal behavior, clustering similar music, discovering patterns in satellite pictures....

You could also work on medical data.

Also, temporal data is interesting to discover trends, repetitive patterns, etc. It could be applied for example to stock market data, etc.

Good luck!

Phil]]> webmasterphilfv The Data Mining Forum Tue, 08 May 2012 21:45:47 -0400 http://forum.ai-directory.com/read.php?5,520,564#msg-564 http://forum.ai-directory.com/read.php?5,520,564#msg-564
>>>> this memory show that how much RAM used by JVM?

The method checkMemory() is for checking how much memory is used by the program running inside the JVM at a particular moment.

>>> And if it is RAM than How it show that max memory is used by program because if program execute all the steps then RAM will be realesed.....

To know what is the max memory, the trick is, as tisonet said, to check the memory several times during the execution. So in my code, it means to call checkMemory() several time during the execution.

For example, consider the PrefixSpan algorithm. You could call the method checkMemory() after each database projection, for example. Then, when the algorithm terminates the variable maxMemory will contain the maximum memory used. It may not be 100 % exact. But it will be very close to the real value.

A better way may be to do as tisonet said and to check the memory before the algorithm starts check regularly an

Beside that, I have one last thing to say about the memory. You probably know that Java use a garbage collector to free some memory when all the memory is used. The garbage collector is "lazy". That means that it will only free memory when all the memory has been used. For example, if you have 1 GB of memory, the garbage collector will start to clean when the 1 GB is full. Why i'm talking about this? It is that if your program reach the point where the garbage collector need to free some memory, it will start to slow down your algorithm and if you are measuring the speed of your algorithm, it will give you some wrong measurements. So, I just want to say to be careful if you reach the limit of memory available on your computer because the garbage collector may interfere with your measurements.

Philippe]]> webmasterphilfv The Data Mining Forum Tue, 08 May 2012 21:33:21 -0400 http://forum.ai-directory.com/read.php?5,520,563#msg-563 http://forum.ai-directory.com/read.php?5,520,563#msg-563
public class MemoryCrawler
{
/////////////////////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////////

private static long _maxMemoryInBytes = 0;
private static long _memoryBeforeRunInBytes = 0;
private static readonly object _mutex = new object();
private readonly Timer _memoryCrawler = new Timer(500);

/////////////////////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////////

public MemoryCrawler()
{
_memoryCrawler.AutoReset = true;
_memoryCrawler.Elapsed += MemoryCrawlerOnElapsed;
}

/////////////////////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////////

private static void MemoryCrawlerOnElapsed(object sender, ElapsedEventArgs elapsedEventArgs)
{
lock (_mutex)
{
long actualMemory = GC.GetTotalMemory(false);
_maxMemoryInBytes = actualMemory > _maxMemoryInBytes ? actualMemory : _maxMemoryInBytes;
}
}

/////////////////////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////////

public void Start()
{
_maxMemoryInBytes = 0;
_memoryBeforeRunInBytes = GC.GetTotalMemory(true);

_memoryCrawler.Start();
}

public void Stop()
{
_memoryCrawler.Stop();
}

public long GetMaxMemoryInMBytes()
{
return (_maxMemoryInBytes - _memoryBeforeRunInBytes) / 1048576;
}

/////////////////////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////////

}]]> tisonet The Data Mining Forum Tue, 08 May 2012 10:37:55 -0400 http://forum.ai-directory.com/read.php?5,520,562#msg-562 http://forum.ai-directory.com/read.php?5,520,562#msg-562
I measure max. memory little bit different.
Before starting mining I take memory snapshot. So I got variable memoryBeforeRun.
Then Iam getting snapshot regularly during mining every x ms. So i got max. memory usage.

Finally max. memory is equal to: memoryBeforeRun - maxMemory.]]> tisonet The Data Mining Forum Tue, 08 May 2012 10:33:14 -0400 http://forum.ai-directory.com/read.php?5,520,561#msg-561 http://forum.ai-directory.com/read.php?5,520,561#msg-561 please let me clear

this memory show that how much RAM used by JVM?

And if it is RAM than How it show that max memory is used by program because if program execute all the steps then RAM will be realesed.....]]> Dvijesh88 The Data Mining Forum Tue, 08 May 2012 07:22:24 -0400 http://forum.ai-directory.com/read.php?5,546,560#msg-560 http://forum.ai-directory.com/read.php?5,546,560#msg-560
Do you wanna interesting data mining topic?
What about data mining in astronomic data, graphs, multimedia or spatio-temporal data? Thats are good enough topic for get Ph. D. at my univerzity.]]> tisonet The Data Mining Forum Tue, 08 May 2012 03:47:42 -0400 http://forum.ai-directory.com/read.php?5,546,559#msg-559 http://forum.ai-directory.com/read.php?5,546,559#msg-559
I would like to make Ph.D. in data mining. maybe clustering research.

It is good topic?

Dawn]]> Dawn The Data Mining Forum Mon, 07 May 2012 23:37:22 -0400 http://forum.ai-directory.com/read.php?5,322,558#msg-558 http://forum.ai-directory.com/read.php?5,322,558#msg-558 [icdm2012.ua.ac.be]
Brussels, Belgium
Conference full paper submissions: June 18, 2012
Paper notifications: September 18, 2012
Format : 6 - 10 pages, IEEE format]]> webmasterphilfv The Data Mining Forum Sun, 06 May 2012 08:37:03 -0400 http://forum.ai-directory.com/read.php?5,549,557#msg-557 http://forum.ai-directory.com/read.php?5,549,557#msg-557 you can ask
give me mail ID, i will sent it to u.]]> Dvijesh88 The Data Mining Forum Wed, 02 May 2012 03:16:23 -0400 http://forum.ai-directory.com/read.php?5,549,556#msg-556 http://forum.ai-directory.com/read.php?5,549,556#msg-556 -------------------------------------------------------
... i sent you that paper.

Can I ask for that paper?

I think that solution for avoiding database scan is use ITEM_IS_EXIST_TABLE which was defined in LAPIN algorithm.]]> tisonet The Data Mining Forum Tue, 01 May 2012 15:55:31 -0400 http://forum.ai-directory.com/read.php?5,322,555#msg-555 http://forum.ai-directory.com/read.php?5,322,555#msg-555 [dtai.cs.kuleuven.be]
Edinburg, July 1st, 2012, Colocated with ICML 2012
Deadline: May 7th
Format: 8 pages (ICML format) or 3 pages (ICML format)]]> webmasterphilfv The Data Mining Forum Tue, 01 May 2012 13:18:16 -0400 http://forum.ai-directory.com/read.php?5,549,554#msg-554 http://forum.ai-directory.com/read.php?5,549,554#msg-554
In my opinion, it would not make sense to use only a vertical database with prefixspan because PrefixSpan need to scan the sequences, which cannot be made efficiently with a vertical database. Vertical databases are better for candidate generation like in ECLAT, SPAM or APRIORITID.

But perhaps that it would be possible to do something like FPGrowth. FPGrowth uses a horizontal database stored into a FPTree. But it also has a header table that allows to find all the sequences that contain an item. This header table could be considered as kind of vertical database.

Perhaps that something like that could be made or has been already made for PrefixSpan. But i'm not sure how this information would be useful... or if it could be used to make the algorithm faster. I did not think a lot about this. :P

Best,

Phlippe]]> webmasterphilfv The Data Mining Forum Tue, 01 May 2012 13:15:20 -0400 http://forum.ai-directory.com/read.php?5,23,553#msg-553 http://forum.ai-directory.com/read.php?5,23,553#msg-553
Please check this page for Java source code for Apriori.

Philippe]]> webmasterphilfv The Data Mining Forum Tue, 01 May 2012 13:05:35 -0400 http://forum.ai-directory.com/read.php?5,23,552#msg-552 http://forum.ai-directory.com/read.php?5,23,552#msg-552 priyanka The Data Mining Forum Tue, 01 May 2012 09:21:32 -0400 http://forum.ai-directory.com/read.php?5,549,551#msg-551 http://forum.ai-directory.com/read.php?5,549,551#msg-551 you are right about PrefixSpan as far i know.

yes that both problem has been solved by one algorithm atleast it claim that they over come both the problem which mention here. i sent you that paper.

so sir cant we represent the dataset in vertical format and apply prefixspan? yeah we have to do some changes in algorithm....
this is just question. and idea i can be wrong.]]> Dvijesh88 The Data Mining Forum Sun, 29 Apr 2012 10:27:34 -0400 http://forum.ai-directory.com/read.php?5,549,550#msg-550 http://forum.ai-directory.com/read.php?5,549,550#msg-550
Here are my quick observations about SPAM and PrefixSpan. Maybe I forgot some elements.

What is good about SPAM:
- it uses a bitmap representation which is memory efficient. There is several optimizations that are posible with bitmaps.
- it uses a vertical representation of the database so that the database only need to be scanned once to create the vertical representation.
- it is very fast to calculate the intersection of two sids sets (sets of sequence ids) by doing a kind of logical AND with two bitmaps.
-. ...

The weak points of SPAM:
- if the sequences are very long, the memory usage will go up because each bitmap take more memory space. If there is a lot of frequent items, more bitmaps will need to be stored into memory.
- SPAM generate candidates and then compute the sids set of the candidate to calculate its support. It may generate many candidates that are not frequent, therefore wasting time. It can also generate candidates that do not appear in the database.
- it is difficult to extend SPAM with additional constraints compared to PrefixSpan
- the database has to be stored in memory
-. ...

What is good about PrefixSpan
- a pattern growth approach. It will not generate candidates that do not appear in the database.
- it includes some optimizations like pseudo-projection, etc.
- it is easy to extend PrefixSpan with additional constraints, for mining closed patterns, etc.
-. ...

What is weak about PrefixSpan
- unlike SPAM, it does not use a vertical representation. So it may need to scan the database several times (even if it uses pseudo-projection).
- the database has to be stored in memory (could be stored on disk, but would be slower)
- ...]]> webmasterphilfv The Data Mining Forum Sat, 28 Apr 2012 18:55:49 -0400 http://forum.ai-directory.com/read.php?5,549,549#msg-549 http://forum.ai-directory.com/read.php?5,549,549#msg-549 where to use these algorithms?]]> Dvijesh88 The Data Mining Forum Sat, 28 Apr 2012 14:08:02 -0400 http://forum.ai-directory.com/read.php?5,546,548#msg-548 http://forum.ai-directory.com/read.php?5,546,548#msg-548 -------------------------------------------------------
> Hi. I'm a student. I want to do a Ph.D. in data
> mining but i don't know which topic is
> interesting. Could you recommend me research
> topics about data mining? I have heard of doing
> decision tree but i think that it is too simple.
> Could you give me a good research topics for doing
> my phd in data mining????]]> Azrs The Data Mining Forum Sat, 28 Apr 2012 02:45:18 -0400 http://forum.ai-directory.com/read.php?5,322,547#msg-547 http://forum.ai-directory.com/read.php?5,322,547#msg-547
BDA 2012 (Bases de Données Avancées 2012)
[bda2012.isima.fr]
23 au 26 oct 2012, Clermont-Ferrand, France
Format: 20 pages A4
Deadline: 7 may 2012
Notification: 12 july 2012]]> webmasterphilfv The Data Mining Forum Fri, 27 Apr 2012 20:53:31 -0400 http://forum.ai-directory.com/read.php?5,546,546#msg-546 http://forum.ai-directory.com/read.php?5,546,546#msg-546 Tenjiku The Data Mining Forum Thu, 26 Apr 2012 00:05:20 -0400 http://forum.ai-directory.com/read.php?5,322,545#msg-545 http://forum.ai-directory.com/read.php?5,322,545#msg-545 Stanford, USA
Date: July 10–13, 2012
Deadline: unknown
Format: unknown
[www.stanford.edu]]]> webmasterphilfv The Data Mining Forum Tue, 24 Apr 2012 19:31:16 -0400