16 Jan 2002, 12:22 UTC
Re: [R] R for large data sets
- From
- Ernesto Jardim ernesto@ipimar.pt
- Date
- Wed, 16 Jan 2002 04:22:29 -0800 (PST)
Hi
I'm using some large datasets and I found the ROracle package to be of
great help.
If you have the chance to create a database in Oracle or MySQL with one
single table for your dataset, you can then use the ROracle package to
access the dataset. I found several advantages on that.
I don't import the data into my environment. I use a small function (see
below) to access the dataset and because the result is a data.frame you
can use it as usually.
Your environment will not be to large and you'll have the ram memory
less full.
It's easier to select subsets with SQL than S/R language.
Hope it helps
Regards
EJ
--//--
ora.fun <- function(){
library(ROracle)
m <- dbManager("Oracle")
con <- dbConnect(m,user="user",password="password")
dat <- quickSQL(con,"select ...")
close(con)
unload(m)
dat
}
--//--
On Tue, 2002-01-15 at 19:43, Prof Brian Ripley wrote:
Quoted or forwarded from earlier messages
> On Tue, 15 Jan 2002, wei, xiaoyan wrote: > > > As a part of our regular data analysis, I have to read in large data sets > > with six columns and about a million rows. In Splus, this usually take a > > couple of minutes. I just tried R, it seems take forever to use read.table() > > to read in the data frame! It did not help much even though I specified > > colClasses and nrows in read.table(). > > > > How is R's ability to analyze large data sets? I used R on solaris 2.6 and I > > used all default compilation flags when building the R package. Will it help > > if I use some compilation flags with higher optimization level? > > It will help to use R-patched, since I guess you are using 1.4.0. > Also, look in the list archives, as I answered this more fully earlier > today. > > In either S-PLUS or R, scan would be a better choice for such a dataset. > > -- > Brian D. Ripley, ripley@stats.ox.ac.uk > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ > University of Oxford, Tel: +44 1865 272861 (self) > 1 South Parks Road, +44 1865 272860 (secr) > Oxford OX1 3TG, UK Fax: +44 1865 272595 > > -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- > r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html > Send "info", "help", or "[un]subscribe" > (in the "body", not the subject !) To: r-help-request@stat.math.ethz.ch > _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._ -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request@stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._