The Data Mining Forum                             open-source data mining software data science journal data mining conferences high utility mining workshop
This forum is about data mining, data science and big data: algorithms, source code, datasets, implementations, optimizations, etc. You are welcome to post call for papers, data mining job ads, link to source code of data mining algorithms or anything else related to data mining. The forum is hosted by P. Fournier-Viger. No registration is required to use this forum!.  
Can I batch process multiple text files?
Posted by: tassieTom
Date: October 18, 2019 01:11PM

Good morning,
I am experimenting with skipgrams and TKS. The results are very impressive in text files that I am using. I am wondering if it is possible to batch process a whole folder of txt files instead of entering the input files one at at time in SPMF. Many thanks, Tom

Options: ReplyQuote
Re: Can I batch process multiple text files?
Date: October 23, 2019 06:31AM

Hi,

I am glad you are getting good results!

Currently this feature is not offered. But since you can call TKS from the command line, you could just write a small script to call SPMF for every file of your folder. If you are using Windows, perhaps that a BAT script could do this. Or if you are on Linux I don't know ;-) Or another could be to write a small program to do a loop on all folder.

It is a good feature, and I will think about how I could add this to a future version of SPMF. Thanks for the suggestion.

Best regards,

Philippe

Options: ReplyQuote
Re: Can I batch process multiple text files?
Date: October 23, 2019 06:31AM

Hi,

I am glad you are getting good results!

Currently this feature is not offered. But since you can call TKS from the command line, you could just write a small script to call SPMF for every file of your folder. If you are using Windows, perhaps that a BAT script could do this. Or if you are on Linux I don't know ;-) Or another could be to write a small program to do a loop on all folder.

It is a good feature, and I will think about how I could add this to a future version of SPMF. Thanks for the suggestion.

Best regards,

Philippe

Options: ReplyQuote
Re: Can I batch process multiple text files?
Posted by: tassieTom
Date: October 23, 2019 08:21PM

Thanks for your response, Philippe. You gave me a clue by mentioning a Bat file for windows. I did a bit of research and managed to get one running which will process all the files in a folder. It works like a charm! Thanks again.

Options: ReplyQuote
Re: Can I batch process multiple text files?
Date: October 24, 2019 08:11AM

Hi

I am glad to know that you have done it successfully with a BAT file!

Would you mind posting the BAT script? Maybe it could be useful to others!

Best regards,

Philippe

Options: ReplyQuote
Re: Can I batch process multiple text files?
Posted by: tassieTom
Date: October 24, 2019 01:13PM

Hi Philippe,
Here is the basic version of a BAT file that will process multiple .text files with TKS in a folder. For simplicity there is no directory checking, so SMPF and the .text files need to be in the same folder.

for %%f in (*.text) do (
echo %%~nf
java -jar spmf.jar run TKS "%%~nf.text" "%%~nfOutput.txt" 50 1 4 "" 1 0
)

That's it!

Hope it's useful to someone, it has saved my day.

By the way, here is a way to rename all your .txt files in a folder to .text with a BAT file too, this script could be added to the beginning of the above script.

ren *.txt *.text

Regards,
Tom

Options: ReplyQuote
Re: Can I batch process multiple text files?
Date: October 25, 2019 08:09PM

Hi Tom,

That is great! Thanks for sharing it!

Best,

Philippe

Options: ReplyQuote


This forum is powered by Phorum and provided by P. Fournier-Viger (© 2012).
Terms of use.