SCM

SCM Repository

[tm] Annotation of /pkg/inst/NEWS.Rd
ViewVC logotype

Annotation of /pkg/inst/NEWS.Rd

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1472 - (view) (download) (as text)

1 : feinerer 1170 \name{NEWS}
2 :     \title{News for Package 'tm'}
3 : khornik 1396 \encoding{UTF-8}
4 : khornik 1472 \section{Changes in tm version 0.7-1}{
5 :     \subsection{BUG FIXES}{
6 :     \itemize{
7 :     \item Compilation fixes for Clang's libc++.
8 :     }
9 :     }
10 :     }
11 : feinerer 1437 \section{Changes in tm version 0.7}{
12 : feinerer 1436 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
13 :     \itemize{
14 :     \item \code{inspect.TermDocumentMatrix()} now displays a sample instead
15 :     of the full matrix. The full dense representation is available via
16 :     \code{as.matrix()}.
17 :     }
18 :     }
19 :     \subsection{NEW FEATURES}{
20 :     \itemize{
21 : feinerer 1437 \item \code{SimpleCorpus} provides a corpus which is optimized for the
22 :     most common usage scenario: importing plain texts from files in a
23 : feinerer 1440 directory or directly from a vector in \R, preprocessing and transforming
24 : feinerer 1437 the texts, and finally exporting them to a term-document matrix. The aim
25 :     is to boost performance and minimize memory pressure. It loads all
26 :     documents into memory, and is designed for medium-sized to large data
27 :     sets.
28 : feinerer 1436 \item \code{inspect()} on text documents as a shorthand for
29 :     \code{writeLines(as.character())}.
30 : feinerer 1450 \item \code{findMostFreqTerms()} finds most frequent terms in a
31 :     document-term or term-document matrix, or a vector of term frequencies.
32 : feinerer 1466 \item \code{tm_parLapply()} is now internally used for the parallelization
33 :     of transformations, filters, and term-document matrix construction. The
34 :     preferred parallelization engine can be registered via
35 :     \code{tm_parLapply_engine()}. The default is to use no parallelization
36 :     (instead of \code{\link[parallel]{mclapply}} (package \pkg{parallel}) in
37 :     previous versions).
38 : feinerer 1436 }
39 :     }
40 :     }
41 : feinerer 1432 \section{Changes in tm version 0.6-2}{
42 :     \subsection{BUG FIXES}{
43 :     \itemize{
44 :     \item \code{format.PlainTextDocument()} now reports only one character
45 :     count for a whole document.
46 :     }
47 :     }
48 :     }
49 : feinerer 1397 \section{Changes in tm version 0.6-1}{
50 : feinerer 1432 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
51 :     \itemize{
52 :     \item \code{format.PlainTextDocument()} now displays a compact
53 :     representation instead of the content. Use \code{as.character()} to
54 :     obtain the character content (which in turn can be applied to a corpus
55 :     via \code{lapply()}).
56 :     }
57 :     }
58 : feinerer 1397 \subsection{NEW FEATURES}{
59 :     \itemize{
60 : feinerer 1425 \item \code{ZipSource()} for processing ZIP files.
61 : feinerer 1397 \item Sources now provide \code{open()} and \code{close()}.
62 : feinerer 1425 \item \code{termFreq()} now accepts \code{Span_Tokenizer} and
63 :     \code{Token_Tokenizer} (both from package \pkg{NLP}) objects as
64 :     tokenizers.
65 :     \item \code{readTagged()}, a reader for text documents containing
66 :     POS-tagged words.
67 : feinerer 1397 }
68 :     }
69 : feinerer 1413 \subsection{BUG FIXES}{
70 :     \itemize{
71 :     \item The function \code{removeWords()} now correctly processes words
72 :     being truncations of others. Reported by Александр Труфанов.
73 :     }
74 :     }
75 : feinerer 1397 }
76 : feinerer 1345 \section{Changes in tm version 0.6}{
77 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
78 :     \itemize{
79 : feinerer 1368 \item \code{DirSource()} and \code{URISource()} now use the argument
80 :     \code{encoding} for conversion via \code{iconv()} to \code{"UTF-8"}.
81 :     \item \code{termFreq()} now uses \code{words()} as the default tokenizer.
82 : feinerer 1345 \item Text documents now provide the functions \code{content()} and
83 :     \code{as.character()} to access the (possibly raw) document content and
84 :     the natural language text in a suitable (not necessarily structured)
85 :     form.
86 : feinerer 1372 \item The internal representation of corpora, sources, and text documents
87 :     changed. Saved objects created with older \pkg{tm} versions are
88 :     incompatible and need to be rebuilt.
89 : feinerer 1345 }
90 :     }
91 :     \subsection{NEW FEATURES}{
92 :     \itemize{
93 :     \item \code{DirSource()} and \code{URISource()} now have a \code{mode}
94 :     argument specifying how elements should be read (no read, binary, text).
95 :     \item Improved high-level documentation on corpora (\code{?Corpus}), text
96 :     documents (\code{?TextDocument}), sources (\code{?Source}), and readers
97 :     (\code{?Reader}).
98 : feinerer 1368 \item Integration with package \pkg{NLP}.
99 :     \item Romanian stopwords. Suggested by Cristian Chirita.
100 :     \item \code{words.PlainTextDocument()} delivers word tokens in the
101 :     document.
102 : feinerer 1345 }
103 :     }
104 :     \subsection{BUG FIXES}{
105 :     \itemize{
106 :     \item The function \code{stemCompletion()} now avoids spurious duplicate
107 :     results. Reported by Seong-Hyeon Kim.
108 :     }
109 :     }
110 :     \subsection{DEPRECATED & DEFUNCT}{
111 :     \itemize{
112 :     \item Following functions have been removed:
113 :     \itemize{
114 :     \item \code{Author()}, \code{DateTimeStamp()}, \code{CMetaData()},
115 :     \code{content_meta()}, \code{DMetaData()}, \code{Description()},
116 :     \code{Heading()}, \code{ID()}, \code{Language()},
117 :     \code{LocalMetaData()}, \code{Origin()}, \code{prescindMeta()},
118 :     \code{sFilter()} (use \code{meta()} instead).
119 :     \item \code{dissimilarity()} (use \code{proxy::dist()} instead).
120 :     \item \code{makeChunks()} (use \code{[} and \code{[[} manually).
121 :     \item \code{summary.Corpus()} and \code{summary.TextRepository()}
122 :     (\code{print()} now gives a more informative but succinct overview).
123 :     \item \code{TextRepository()} and \code{RepoMetaData()} (use e.g. a
124 :     list to store multiple corpora instead).
125 :     }
126 :     }
127 :     }
128 :     }
129 : feinerer 1239 \section{Changes in tm version 0.5-10}{
130 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
131 :     \itemize{
132 :     \item License changed to GPL-3 (from GPL-2 | GPL-3).
133 : feinerer 1255 \item Following functions have been renamed:
134 :     \itemize{
135 : feinerer 1432 \item \code{tm_tag_score()} to \code{tm_term_score()}.
136 : feinerer 1255 }
137 : feinerer 1239 }
138 :     }
139 : feinerer 1242 \subsection{DEPRECATED & DEFUNCT}{
140 :     \itemize{
141 : feinerer 1253 \item Following functions have been removed:
142 :     \itemize{
143 : feinerer 1277 \item \code{Dictionary()} (use a character vector instead; use
144 : feinerer 1280 \code{Terms()} to extract terms from a document-term or term-document
145 : feinerer 1277 matrix),
146 : feinerer 1258 \item \code{GmaneSource()} (but still available via an example in
147 : feinerer 1283 \code{XMLSource()}),
148 : feinerer 1260 \item \code{preprocessReut21578XML()} (moved to package
149 :     \pkg{tm.corpus.Reuters21578}),
150 : feinerer 1258 \item \code{readGmane()} (but still available via an example in
151 : feinerer 1283 \code{readXML()}),
152 : feinerer 1432 \item \code{searchFullText()} and \code{tm_intersect()}
153 : feinerer 1253 (use \code{grep()} instead).
154 :     }
155 : feinerer 1242 \item Following S3 classes are no longer registered as S4 classes:
156 :     \itemize{
157 : feinerer 1432 \item \code{VCorpus} and \code{PlainTextDocument}.
158 : feinerer 1242 }
159 :     }
160 :     }
161 : feinerer 1239 }
162 : feinerer 1224 \section{Changes in tm version 0.5-9}{
163 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
164 :     \itemize{
165 : feinerer 1226 \item Stemming functionality is now provided by the package
166 :     \pkg{SnowballC} replacing packages \pkg{Snowball} and \pkg{RWeka}.
167 : feinerer 1224 \item All stopword lists (besides Catalan and SMART) available via
168 :     \code{stopwords()} now come from the Snowball stemmer project.
169 : feinerer 1227 \item Transformations, filters, and term-document matrix construction
170 :     now use \code{\link[parallel]{mclapply}} (package \pkg{parallel}).
171 :     Packages \pkg{snow} and \pkg{Rmpi} are no longer used.
172 : feinerer 1224 }
173 :     }
174 : feinerer 1227 \subsection{DEPRECATED & DEFUNCT}{
175 :     \itemize{
176 :     \item Following functions have been removed:
177 :     \itemize{
178 : feinerer 1432 \item \code{tm_startCluster()} and \code{tm_stopCluster()}.
179 : feinerer 1227 }
180 :     }
181 :     }
182 : feinerer 1224 }
183 : feinerer 1173 \section{Changes in tm version 0.5-8}{
184 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
185 :     \itemize{
186 :     \item The function \code{termFreq()} now processes the
187 :     \code{tolower} and \code{tokenize} options first.
188 :     }
189 :     }
190 : feinerer 1174 \subsection{NEW FEATURES}{
191 :     \itemize{
192 :     \item Catalan stopwords. Requested by Xavier Fernández i Marín.
193 :     }
194 :     }
195 : feinerer 1173 \subsection{BUG FIXES}{
196 :     \itemize{
197 :     \item The function \code{termFreq()} now correctly accepts
198 :     user-provided stopwords. Reported by Bettina Grün.
199 :     \item The function \code{termFreq()} now correctly handles the
200 :     lower bound of the option \code{wordLength}. Reported by Steven
201 :     C. Bagley.
202 :     }
203 :     }
204 :     }
205 : feinerer 1170 \section{Changes in tm version 0.5-7}{
206 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
207 :     \itemize{
208 :     \item The function \code{termFreq()} provides two new arguments for
209 :     generalized bounds checking of term frequencies and word
210 :     lengths. This replaces the arguments minDocFreq and
211 :     minWordLength.
212 :     \item The function \code{termFreq()} is now sensitive to the order of
213 :     control options.
214 :     }
215 :     }
216 :     \subsection{NEW FEATURES}{
217 :     \itemize{
218 :     \item Weighting schemata for term-document matrices in SMART notation.
219 :     \item Local and global options for term-document matrix
220 :     construction.
221 :     \item SMART stopword list was added.
222 :     }
223 :     }
224 :     }
225 :     \section{Changes in tm version 0.5-5}{
226 :     \subsection{NEW FEATURES}{
227 :     \itemize{
228 :     \item Access documents in a corpus by names (fallback to IDs if names are
229 :     not set), i.e., allow a string for the corpus operator `[[`.
230 :     }
231 :     }
232 :     \subsection{BUG FIXES}{
233 :     \itemize{
234 :     \item The function \code{findFreqTerms()} now checks bounds on a global level
235 :     (to comply with the manual page) instead per document. Reported
236 :     and fixed by Thomas Zapf-Schramm.
237 :     }
238 :     }
239 :     }
240 :     \section{Changes in tm version 0.5-4}{
241 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
242 :     \itemize{
243 :     \item Use IETF language tags for language codes (instead of ISO 639-2).
244 :     }
245 :     }
246 :     \subsection{NEW FEATURES}{
247 :     \itemize{
248 :     \item The function \code{tm_tag_score()} provides functionality to score
249 :     documents based on the number of tags found. This is useful for
250 :     sentiment analysis.
251 :     \item The weighting function for term frequency-inverse document
252 :     frequency \code{weightTfIdf()} has now an option for term
253 :     normalization.
254 :     \item Plotting functions to test for Zipf's and Heaps' law on a
255 :     term-document matrix were added: \code{Zipf_plot()} and
256 :     \code{Heaps_plot()}. Contributed by Kurt Hornik.
257 :     }
258 :     }
259 :     }
260 :     \section{Changes in tm version 0.5-3}{
261 :     \subsection{NEW FEATURES}{
262 :     \itemize{
263 :     \item The reader function \code{readRCV1asPlain()} was added and combines the
264 :     functionality of \code{readRCV1()} and \code{as.PlainTextDocument()}.
265 :     \item The function \code{stemCompletion()} has a set of new heuristics.
266 :     }
267 :     }
268 :     }
269 :     \section{Changes in tm version 0.5-2}{
270 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
271 :     \itemize{
272 :     \item The function \code{termFreq()} which is used for building a
273 :     term-document matrix now uses a whitespace oriented tokenizer
274 :     as default.
275 :     }
276 :     }
277 :     \subsection{NEW FEATURES}{
278 :     \itemize{
279 :     \item A combine method for merging multiple term-document matrices
280 :     was added (\code{c.TermDocumentMatrix()}).
281 :     \item The function \code{termFreq()} has now an option to remove
282 :     punctuation characters.
283 :     }
284 :     }
285 :     \subsection{DEPRECATED & DEFUNCT}{
286 :     \itemize{
287 :     \item Following functions have been removed:
288 :     \itemize{
289 : feinerer 1432 \item \code{CSVSource()} (use \code{DataframeSource(read.csv(..., stringsAsFactors = FALSE))} instead), and
290 : feinerer 1170 \item \code{TermDocMatrix()} (use \code{DocumentTermMatrix()} instead).
291 :     }
292 :     }
293 :     }
294 :     \subsection{BUG FIXES}{
295 :     \itemize{
296 :     \item \code{removeWords()} no longer skips words at the beginning or the end
297 :     of a line. Reported by Mark Kimpel.
298 :     }
299 :     }
300 :     }
301 :     \section{Changes in tm version 0.5-1}{
302 :     \subsection{BUG FIXES}{
303 :     \itemize{
304 :     \item \code{preprocessReut21578XML()} no longer generates invalid file names.
305 :     }
306 :     }
307 :     }
308 :     \section{Changes in tm version 0.5}{
309 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
310 :     \itemize{
311 :     \item All classes, functions, and generics are reimplemented using
312 :     the S3 class system.
313 :     \item Following functions have been renamed:
314 :     \itemize{
315 : feinerer 1432 \item \code{activateCluster()} to \code{tm_startCluster()},
316 : feinerer 1170 \item \code{asPlain()} to \code{as.PlainTextDocument()},
317 :     \item \code{deactivateCluster()} to \code{tm_stopCluster()},
318 :     \item \code{tmFilter()} to \code{tm_filter()},
319 :     \item \code{tmIndex()} to \code{tm_index()},
320 :     \item \code{tmIntersect()} to \code{tm_intersect()}, and
321 :     \item \code{tmMap()} to \code{tm_map()}.
322 :     }
323 :     \item Mail handling functionality is factored out to the
324 : feinerer 1233 \pkg{tm.plugin.mail} package.
325 : feinerer 1170 }
326 :     }
327 :     \subsection{DEPRECATED & DEFUNCT}{
328 :     \itemize{
329 :     \item Following functions have been removed:
330 :     \itemize{
331 :     \item \code{tmTolower()} (use \code{tolower()} instead), and
332 : feinerer 1172 \item \code{replacePatterns()} (use \code{gsub()} instead).
333 : feinerer 1170 }
334 :     }
335 :     }
336 :     }
337 :     \section{Changes in tm version 0.4}{
338 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
339 :     \itemize{
340 :     \item The Corpus class is now virtual providing an abstract
341 :     interface.
342 :     \item VCorpus, the default implementation of the abstract corpus
343 :     interface (by subclassing), provides a corpus with volatile (=
344 : feinerer 1438 standard \R object) semantics. It loads all documents into
345 : feinerer 1170 memory, and is designed for small to medium-sized data sets.
346 :     \item PCorpus, an implementation of the abstract corpus interface (by
347 :     subclassing), provides a corpus with permanent storage
348 :     semantics. The actual data is stored in an external database
349 : feinerer 1233 (file) object (as supported by the \pkg{filehash} package), with
350 : feinerer 1170 automatic (un-)loading into memory. It is designed for systems
351 :     with small memory.
352 :     \item Language codes are now in ISO 639-2 (instead of ISO 639-1).
353 :     \item Reader functions no longer have a load argument for lazy
354 :     loading.
355 :     }
356 :     }
357 :     \subsection{NEW FEATURES}{
358 :     \itemize{
359 :     \item The reader function \code{readReut21578XMLasPlain()} was added and
360 :     combines the functionality of \code{readReut21578XML()} and \code{asPlain()}.
361 :     }
362 :     }
363 :     \subsection{BUG FIXES}{
364 :     \itemize{
365 :     \item \code{weightTfIdf()} no longer applies a binary weighting to an input
366 :     matrix in term frequency format (which happened only in 0.3-4).
367 :     }
368 :     }
369 :     }
370 :     \section{Changes in tm version 0.3-4}{
371 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
372 :     \itemize{
373 :     \item \code{.onLoad()} no longer tries to start a MPI cluster (which often
374 :     failed in misconfigured environments). Use \code{activateCluster()}
375 :     and \code{deactivateCluster()} instead.
376 :     \item DocumentTermMatrix (the improved reimplementation of defunct
377 : feinerer 1233 TermDocMatrix) does not use the \pkg{Matrix} package anymore.
378 : feinerer 1170 }
379 :     }
380 :     \subsection{NEW FEATURES}{
381 :     \itemize{
382 :     \item The \code{DirSource()} constructor now accepts the two new (optional)
383 :     arguments pattern and ignore.case. With pattern one can define
384 :     a regular expression for selecting only matching files, and
385 :     ignore.case specifies whether pattern-matching is
386 :     case-sensitive.
387 :     \item The \code{readNewsgroup()} reader function can now be configured for
388 :     custom date formats (via the DateFormat argument).
389 :     \item The \code{readPDF()} reader function can now be configured (via the
390 :     PdfinfoOptions and PdftotextOptions arguments).
391 :     \item The \code{readDOC()} reader function can now be configured (via the
392 :     AntiwordOptions argument).
393 :     \item Sources now can be vectorized. This allows faster corpus
394 :     construction.
395 :     \item New XMLSource class for arbitrary XML files.
396 :     \item The new \code{readTabular()} reader function allows to create a custom
397 :     tailor-made reader configured via mappings from a tabular data
398 :     structure.
399 :     \item The new \code{readXML()} reader function allows to read in arbitrary
400 :     XML files which are described with a specification.
401 :     \item The new \code{tmReduce()} transformation allows to combine multiple
402 :     maps into one transformation.
403 :     }
404 :     }
405 :     \subsection{DEPRECATED & DEFUNCT}{
406 :     \itemize{
407 :     \item CSVSource is defunct (use DataframeSource instead).
408 :     \item weightLogical is defunct.
409 :     \item TermDocMatrix is defunct (use DocumentTermMatrix or
410 :     TermDocumentMatrix instead).
411 :     }
412 :     }
413 :     }
414 :     \section{Changes in tm version 0.3-3}{
415 :     \subsection{NEW FEATURES}{
416 :     \itemize{
417 :     \item The abstract Source class gets a default implementation for
418 :     the \code{stepNext()} method. It increments the position counter by
419 :     one, a reasonable value for most sources. For special purposes
420 :     custom methods can be created via overloading \code{stepNext()} of
421 :     the subclass.
422 :     \item New URISource class for a single document identified by a
423 :     Uniform Resource Identifier.
424 :     \item New DataframeSource for documents stored in a data frame. Each
425 :     row is interpreted as a single document.
426 :     }
427 :     }
428 :     \subsection{BUG FIXES}{
429 :     \itemize{
430 :     \item Fix off-by-one error in \code{convertMboxEml()} function. Reported by
431 :     Angela Bohn.
432 :     \item Sort row indices in sparse term-document matrices. Kudos to
433 :     Martin Mächler for his suggestions.
434 :     \item Sources and readers no longer evaluate calls in a non-standard
435 :     way.
436 :     }
437 :     }
438 :     }
439 :     \section{Changes in tm version 0.3-2}{
440 :     \subsection{NEW FEATURES}{
441 :     \itemize{
442 :     \item Weighting functions now have an Acronym slot containing
443 :     abbreviations of the weighting functions' names. This is highly
444 :     useful when generating tables with indications which weighting
445 :     scheme was actually used for your experiments.
446 :     \item The functions \code{tmFilter()}, \code{tmIndex()}, \code{tmMap()} and \code{TermDocMatrix()}
447 : feinerer 1233 now can use a MPI cluster (via the \pkg{snow} and \pkg{Rmpi} packages) if
448 : feinerer 1170 available. Use \code{(de)activateCluster()} to manually override
449 :     cluster usage settings. Special thanks to Stefan Theussl for
450 :     his constructive comments.
451 :     \item The Source class receives a new Length slot. It contains the
452 :     number of elements provided by the source (although there
453 :     might be rare cases where the number cannot be determined in
454 :     advance---then it should be set to zero).
455 :     }
456 :     }
457 :     }

R-Forge@R-project.org
ViewVC Help
Powered by ViewVC 1.0.0  
Thanks to:
Vienna University of Economics and Business University of Wisconsin - Madison Powered By FusionForge