SCM

SCM Repository

[tm] Annotation of /pkg/inst/NEWS.Rd
ViewVC logotype

Annotation of /pkg/inst/NEWS.Rd

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1520 - (view) (download) (as text)

1 : feinerer 1170 \name{NEWS}
2 :     \title{News for Package 'tm'}
3 : khornik 1396 \encoding{UTF-8}
4 : feinerer 1520 \section{Changes in tm version 0.7-3}{
5 :     \subsection{BUG FIXES}{
6 :     \itemize{
7 :     \item \code{scan_tokenizer()} now works with character vectors and
8 :     character strings.
9 :     \item \code{removePunctuation()} now works again in \code{latin1} locales.
10 :     \item Handle empty term-document matrices gracefully.
11 :     }
12 :     }
13 :     }
14 : feinerer 1474 \section{Changes in tm version 0.7-2}{
15 : feinerer 1481 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
16 :     \itemize{
17 :     \item \code{DataframeSource} now only processes data frames with the two
18 :     mandatory columns \code{"doc_id"} and \code{"text"}. Additional columns
19 :     are used as document level metadata. This implements compatibility with
20 :     \emph{Text Interchange Formats} corpora
21 :     (\url{https://github.com/ropensci/tif}).
22 :     \item \code{readTabular()} has been removed. Use \code{DataframeSource}
23 :     instead.
24 : feinerer 1495 \item \code{removeNumbers()} and \code{removePunctuation()} now have an
25 :     argument \code{ucp} to check for Unicode general categories \code{Nd}
26 :     (decimal digits) and \code{P} (punctuation), respectively. Contributed
27 :     by Kurt Hornik.
28 : feinerer 1503 \item The package \pkg{xml2} is now imported for \acronym{XML}
29 :     functionality instead of the (\acronym{CRAN} maintainer orphaned)
30 :     package \pkg{XML}.
31 : feinerer 1481 }
32 :     }
33 : feinerer 1488 \subsection{NEW FEATURES}{
34 :     \itemize{
35 :     \item \code{Boost_tokenizer} provides a tokenizer based on the Boost
36 :     (\url{http://www.boost.org}) Tokenizer.
37 :     }
38 :     }
39 : feinerer 1474 \subsection{BUG FIXES}{
40 :     \itemize{
41 :     \item Correctly handle the \code{dictionary} argument when constructing a
42 : feinerer 1478 term-document matrix from a \code{SimpleCorpus} (reported by Joe
43 :     Corrigan) or from a \code{VCorpus} (reported by Mark Rosenstein).
44 : feinerer 1474 }
45 :     }
46 :     }
47 : khornik 1472 \section{Changes in tm version 0.7-1}{
48 :     \subsection{BUG FIXES}{
49 :     \itemize{
50 :     \item Compilation fixes for Clang's libc++.
51 :     }
52 :     }
53 :     }
54 : feinerer 1437 \section{Changes in tm version 0.7}{
55 : feinerer 1436 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
56 :     \itemize{
57 :     \item \code{inspect.TermDocumentMatrix()} now displays a sample instead
58 :     of the full matrix. The full dense representation is available via
59 :     \code{as.matrix()}.
60 :     }
61 :     }
62 :     \subsection{NEW FEATURES}{
63 :     \itemize{
64 : feinerer 1437 \item \code{SimpleCorpus} provides a corpus which is optimized for the
65 :     most common usage scenario: importing plain texts from files in a
66 : feinerer 1440 directory or directly from a vector in \R, preprocessing and transforming
67 : feinerer 1437 the texts, and finally exporting them to a term-document matrix. The aim
68 :     is to boost performance and minimize memory pressure. It loads all
69 :     documents into memory, and is designed for medium-sized to large data
70 :     sets.
71 : feinerer 1436 \item \code{inspect()} on text documents as a shorthand for
72 :     \code{writeLines(as.character())}.
73 : feinerer 1450 \item \code{findMostFreqTerms()} finds most frequent terms in a
74 :     document-term or term-document matrix, or a vector of term frequencies.
75 : feinerer 1466 \item \code{tm_parLapply()} is now internally used for the parallelization
76 :     of transformations, filters, and term-document matrix construction. The
77 :     preferred parallelization engine can be registered via
78 :     \code{tm_parLapply_engine()}. The default is to use no parallelization
79 :     (instead of \code{\link[parallel]{mclapply}} (package \pkg{parallel}) in
80 :     previous versions).
81 : feinerer 1436 }
82 :     }
83 :     }
84 : feinerer 1432 \section{Changes in tm version 0.6-2}{
85 :     \subsection{BUG FIXES}{
86 :     \itemize{
87 :     \item \code{format.PlainTextDocument()} now reports only one character
88 :     count for a whole document.
89 :     }
90 :     }
91 :     }
92 : feinerer 1397 \section{Changes in tm version 0.6-1}{
93 : feinerer 1432 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
94 :     \itemize{
95 :     \item \code{format.PlainTextDocument()} now displays a compact
96 :     representation instead of the content. Use \code{as.character()} to
97 :     obtain the character content (which in turn can be applied to a corpus
98 :     via \code{lapply()}).
99 :     }
100 :     }
101 : feinerer 1397 \subsection{NEW FEATURES}{
102 :     \itemize{
103 : feinerer 1425 \item \code{ZipSource()} for processing ZIP files.
104 : feinerer 1397 \item Sources now provide \code{open()} and \code{close()}.
105 : feinerer 1425 \item \code{termFreq()} now accepts \code{Span_Tokenizer} and
106 :     \code{Token_Tokenizer} (both from package \pkg{NLP}) objects as
107 :     tokenizers.
108 :     \item \code{readTagged()}, a reader for text documents containing
109 :     POS-tagged words.
110 : feinerer 1397 }
111 :     }
112 : feinerer 1413 \subsection{BUG FIXES}{
113 :     \itemize{
114 :     \item The function \code{removeWords()} now correctly processes words
115 :     being truncations of others. Reported by Александр Труфанов.
116 :     }
117 :     }
118 : feinerer 1397 }
119 : feinerer 1345 \section{Changes in tm version 0.6}{
120 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
121 :     \itemize{
122 : feinerer 1368 \item \code{DirSource()} and \code{URISource()} now use the argument
123 :     \code{encoding} for conversion via \code{iconv()} to \code{"UTF-8"}.
124 :     \item \code{termFreq()} now uses \code{words()} as the default tokenizer.
125 : feinerer 1345 \item Text documents now provide the functions \code{content()} and
126 :     \code{as.character()} to access the (possibly raw) document content and
127 :     the natural language text in a suitable (not necessarily structured)
128 :     form.
129 : feinerer 1372 \item The internal representation of corpora, sources, and text documents
130 :     changed. Saved objects created with older \pkg{tm} versions are
131 :     incompatible and need to be rebuilt.
132 : feinerer 1345 }
133 :     }
134 :     \subsection{NEW FEATURES}{
135 :     \itemize{
136 :     \item \code{DirSource()} and \code{URISource()} now have a \code{mode}
137 :     argument specifying how elements should be read (no read, binary, text).
138 :     \item Improved high-level documentation on corpora (\code{?Corpus}), text
139 :     documents (\code{?TextDocument}), sources (\code{?Source}), and readers
140 :     (\code{?Reader}).
141 : feinerer 1368 \item Integration with package \pkg{NLP}.
142 :     \item Romanian stopwords. Suggested by Cristian Chirita.
143 :     \item \code{words.PlainTextDocument()} delivers word tokens in the
144 :     document.
145 : feinerer 1345 }
146 :     }
147 :     \subsection{BUG FIXES}{
148 :     \itemize{
149 :     \item The function \code{stemCompletion()} now avoids spurious duplicate
150 :     results. Reported by Seong-Hyeon Kim.
151 :     }
152 :     }
153 :     \subsection{DEPRECATED & DEFUNCT}{
154 :     \itemize{
155 :     \item Following functions have been removed:
156 :     \itemize{
157 :     \item \code{Author()}, \code{DateTimeStamp()}, \code{CMetaData()},
158 :     \code{content_meta()}, \code{DMetaData()}, \code{Description()},
159 :     \code{Heading()}, \code{ID()}, \code{Language()},
160 :     \code{LocalMetaData()}, \code{Origin()}, \code{prescindMeta()},
161 :     \code{sFilter()} (use \code{meta()} instead).
162 :     \item \code{dissimilarity()} (use \code{proxy::dist()} instead).
163 :     \item \code{makeChunks()} (use \code{[} and \code{[[} manually).
164 :     \item \code{summary.Corpus()} and \code{summary.TextRepository()}
165 :     (\code{print()} now gives a more informative but succinct overview).
166 :     \item \code{TextRepository()} and \code{RepoMetaData()} (use e.g. a
167 :     list to store multiple corpora instead).
168 :     }
169 :     }
170 :     }
171 :     }
172 : feinerer 1239 \section{Changes in tm version 0.5-10}{
173 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
174 :     \itemize{
175 :     \item License changed to GPL-3 (from GPL-2 | GPL-3).
176 : feinerer 1255 \item Following functions have been renamed:
177 :     \itemize{
178 : feinerer 1432 \item \code{tm_tag_score()} to \code{tm_term_score()}.
179 : feinerer 1255 }
180 : feinerer 1239 }
181 :     }
182 : feinerer 1242 \subsection{DEPRECATED & DEFUNCT}{
183 :     \itemize{
184 : feinerer 1253 \item Following functions have been removed:
185 :     \itemize{
186 : feinerer 1277 \item \code{Dictionary()} (use a character vector instead; use
187 : feinerer 1280 \code{Terms()} to extract terms from a document-term or term-document
188 : feinerer 1277 matrix),
189 : feinerer 1258 \item \code{GmaneSource()} (but still available via an example in
190 : feinerer 1283 \code{XMLSource()}),
191 : feinerer 1260 \item \code{preprocessReut21578XML()} (moved to package
192 :     \pkg{tm.corpus.Reuters21578}),
193 : feinerer 1258 \item \code{readGmane()} (but still available via an example in
194 : feinerer 1283 \code{readXML()}),
195 : feinerer 1432 \item \code{searchFullText()} and \code{tm_intersect()}
196 : feinerer 1253 (use \code{grep()} instead).
197 :     }
198 : feinerer 1242 \item Following S3 classes are no longer registered as S4 classes:
199 :     \itemize{
200 : feinerer 1432 \item \code{VCorpus} and \code{PlainTextDocument}.
201 : feinerer 1242 }
202 :     }
203 :     }
204 : feinerer 1239 }
205 : feinerer 1224 \section{Changes in tm version 0.5-9}{
206 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
207 :     \itemize{
208 : feinerer 1226 \item Stemming functionality is now provided by the package
209 :     \pkg{SnowballC} replacing packages \pkg{Snowball} and \pkg{RWeka}.
210 : feinerer 1224 \item All stopword lists (besides Catalan and SMART) available via
211 :     \code{stopwords()} now come from the Snowball stemmer project.
212 : feinerer 1227 \item Transformations, filters, and term-document matrix construction
213 :     now use \code{\link[parallel]{mclapply}} (package \pkg{parallel}).
214 :     Packages \pkg{snow} and \pkg{Rmpi} are no longer used.
215 : feinerer 1224 }
216 :     }
217 : feinerer 1227 \subsection{DEPRECATED & DEFUNCT}{
218 :     \itemize{
219 :     \item Following functions have been removed:
220 :     \itemize{
221 : feinerer 1432 \item \code{tm_startCluster()} and \code{tm_stopCluster()}.
222 : feinerer 1227 }
223 :     }
224 :     }
225 : feinerer 1224 }
226 : feinerer 1173 \section{Changes in tm version 0.5-8}{
227 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
228 :     \itemize{
229 :     \item The function \code{termFreq()} now processes the
230 :     \code{tolower} and \code{tokenize} options first.
231 :     }
232 :     }
233 : feinerer 1174 \subsection{NEW FEATURES}{
234 :     \itemize{
235 :     \item Catalan stopwords. Requested by Xavier Fernández i Marín.
236 :     }
237 :     }
238 : feinerer 1173 \subsection{BUG FIXES}{
239 :     \itemize{
240 :     \item The function \code{termFreq()} now correctly accepts
241 :     user-provided stopwords. Reported by Bettina Grün.
242 :     \item The function \code{termFreq()} now correctly handles the
243 :     lower bound of the option \code{wordLength}. Reported by Steven
244 :     C. Bagley.
245 :     }
246 :     }
247 :     }
248 : feinerer 1170 \section{Changes in tm version 0.5-7}{
249 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
250 :     \itemize{
251 :     \item The function \code{termFreq()} provides two new arguments for
252 :     generalized bounds checking of term frequencies and word
253 :     lengths. This replaces the arguments minDocFreq and
254 :     minWordLength.
255 :     \item The function \code{termFreq()} is now sensitive to the order of
256 :     control options.
257 :     }
258 :     }
259 :     \subsection{NEW FEATURES}{
260 :     \itemize{
261 :     \item Weighting schemata for term-document matrices in SMART notation.
262 :     \item Local and global options for term-document matrix
263 :     construction.
264 :     \item SMART stopword list was added.
265 :     }
266 :     }
267 :     }
268 :     \section{Changes in tm version 0.5-5}{
269 :     \subsection{NEW FEATURES}{
270 :     \itemize{
271 :     \item Access documents in a corpus by names (fallback to IDs if names are
272 :     not set), i.e., allow a string for the corpus operator `[[`.
273 :     }
274 :     }
275 :     \subsection{BUG FIXES}{
276 :     \itemize{
277 :     \item The function \code{findFreqTerms()} now checks bounds on a global level
278 :     (to comply with the manual page) instead per document. Reported
279 :     and fixed by Thomas Zapf-Schramm.
280 :     }
281 :     }
282 :     }
283 :     \section{Changes in tm version 0.5-4}{
284 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
285 :     \itemize{
286 :     \item Use IETF language tags for language codes (instead of ISO 639-2).
287 :     }
288 :     }
289 :     \subsection{NEW FEATURES}{
290 :     \itemize{
291 :     \item The function \code{tm_tag_score()} provides functionality to score
292 :     documents based on the number of tags found. This is useful for
293 :     sentiment analysis.
294 :     \item The weighting function for term frequency-inverse document
295 :     frequency \code{weightTfIdf()} has now an option for term
296 :     normalization.
297 :     \item Plotting functions to test for Zipf's and Heaps' law on a
298 :     term-document matrix were added: \code{Zipf_plot()} and
299 :     \code{Heaps_plot()}. Contributed by Kurt Hornik.
300 :     }
301 :     }
302 :     }
303 :     \section{Changes in tm version 0.5-3}{
304 :     \subsection{NEW FEATURES}{
305 :     \itemize{
306 :     \item The reader function \code{readRCV1asPlain()} was added and combines the
307 :     functionality of \code{readRCV1()} and \code{as.PlainTextDocument()}.
308 :     \item The function \code{stemCompletion()} has a set of new heuristics.
309 :     }
310 :     }
311 :     }
312 :     \section{Changes in tm version 0.5-2}{
313 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
314 :     \itemize{
315 :     \item The function \code{termFreq()} which is used for building a
316 :     term-document matrix now uses a whitespace oriented tokenizer
317 :     as default.
318 :     }
319 :     }
320 :     \subsection{NEW FEATURES}{
321 :     \itemize{
322 :     \item A combine method for merging multiple term-document matrices
323 :     was added (\code{c.TermDocumentMatrix()}).
324 :     \item The function \code{termFreq()} has now an option to remove
325 :     punctuation characters.
326 :     }
327 :     }
328 :     \subsection{DEPRECATED & DEFUNCT}{
329 :     \itemize{
330 :     \item Following functions have been removed:
331 :     \itemize{
332 : feinerer 1432 \item \code{CSVSource()} (use \code{DataframeSource(read.csv(..., stringsAsFactors = FALSE))} instead), and
333 : feinerer 1170 \item \code{TermDocMatrix()} (use \code{DocumentTermMatrix()} instead).
334 :     }
335 :     }
336 :     }
337 :     \subsection{BUG FIXES}{
338 :     \itemize{
339 :     \item \code{removeWords()} no longer skips words at the beginning or the end
340 :     of a line. Reported by Mark Kimpel.
341 :     }
342 :     }
343 :     }
344 :     \section{Changes in tm version 0.5-1}{
345 :     \subsection{BUG FIXES}{
346 :     \itemize{
347 :     \item \code{preprocessReut21578XML()} no longer generates invalid file names.
348 :     }
349 :     }
350 :     }
351 :     \section{Changes in tm version 0.5}{
352 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
353 :     \itemize{
354 :     \item All classes, functions, and generics are reimplemented using
355 :     the S3 class system.
356 :     \item Following functions have been renamed:
357 :     \itemize{
358 : feinerer 1432 \item \code{activateCluster()} to \code{tm_startCluster()},
359 : feinerer 1170 \item \code{asPlain()} to \code{as.PlainTextDocument()},
360 :     \item \code{deactivateCluster()} to \code{tm_stopCluster()},
361 :     \item \code{tmFilter()} to \code{tm_filter()},
362 :     \item \code{tmIndex()} to \code{tm_index()},
363 :     \item \code{tmIntersect()} to \code{tm_intersect()}, and
364 :     \item \code{tmMap()} to \code{tm_map()}.
365 :     }
366 :     \item Mail handling functionality is factored out to the
367 : feinerer 1233 \pkg{tm.plugin.mail} package.
368 : feinerer 1170 }
369 :     }
370 :     \subsection{DEPRECATED & DEFUNCT}{
371 :     \itemize{
372 :     \item Following functions have been removed:
373 :     \itemize{
374 :     \item \code{tmTolower()} (use \code{tolower()} instead), and
375 : feinerer 1172 \item \code{replacePatterns()} (use \code{gsub()} instead).
376 : feinerer 1170 }
377 :     }
378 :     }
379 :     }
380 :     \section{Changes in tm version 0.4}{
381 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
382 :     \itemize{
383 :     \item The Corpus class is now virtual providing an abstract
384 :     interface.
385 :     \item VCorpus, the default implementation of the abstract corpus
386 :     interface (by subclassing), provides a corpus with volatile (=
387 : feinerer 1438 standard \R object) semantics. It loads all documents into
388 : feinerer 1170 memory, and is designed for small to medium-sized data sets.
389 :     \item PCorpus, an implementation of the abstract corpus interface (by
390 :     subclassing), provides a corpus with permanent storage
391 :     semantics. The actual data is stored in an external database
392 : feinerer 1233 (file) object (as supported by the \pkg{filehash} package), with
393 : feinerer 1170 automatic (un-)loading into memory. It is designed for systems
394 :     with small memory.
395 :     \item Language codes are now in ISO 639-2 (instead of ISO 639-1).
396 :     \item Reader functions no longer have a load argument for lazy
397 :     loading.
398 :     }
399 :     }
400 :     \subsection{NEW FEATURES}{
401 :     \itemize{
402 :     \item The reader function \code{readReut21578XMLasPlain()} was added and
403 :     combines the functionality of \code{readReut21578XML()} and \code{asPlain()}.
404 :     }
405 :     }
406 :     \subsection{BUG FIXES}{
407 :     \itemize{
408 :     \item \code{weightTfIdf()} no longer applies a binary weighting to an input
409 :     matrix in term frequency format (which happened only in 0.3-4).
410 :     }
411 :     }
412 :     }
413 :     \section{Changes in tm version 0.3-4}{
414 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
415 :     \itemize{
416 :     \item \code{.onLoad()} no longer tries to start a MPI cluster (which often
417 :     failed in misconfigured environments). Use \code{activateCluster()}
418 :     and \code{deactivateCluster()} instead.
419 :     \item DocumentTermMatrix (the improved reimplementation of defunct
420 : feinerer 1233 TermDocMatrix) does not use the \pkg{Matrix} package anymore.
421 : feinerer 1170 }
422 :     }
423 :     \subsection{NEW FEATURES}{
424 :     \itemize{
425 :     \item The \code{DirSource()} constructor now accepts the two new (optional)
426 :     arguments pattern and ignore.case. With pattern one can define
427 :     a regular expression for selecting only matching files, and
428 :     ignore.case specifies whether pattern-matching is
429 :     case-sensitive.
430 :     \item The \code{readNewsgroup()} reader function can now be configured for
431 :     custom date formats (via the DateFormat argument).
432 :     \item The \code{readPDF()} reader function can now be configured (via the
433 :     PdfinfoOptions and PdftotextOptions arguments).
434 :     \item The \code{readDOC()} reader function can now be configured (via the
435 :     AntiwordOptions argument).
436 :     \item Sources now can be vectorized. This allows faster corpus
437 :     construction.
438 :     \item New XMLSource class for arbitrary XML files.
439 :     \item The new \code{readTabular()} reader function allows to create a custom
440 :     tailor-made reader configured via mappings from a tabular data
441 :     structure.
442 :     \item The new \code{readXML()} reader function allows to read in arbitrary
443 :     XML files which are described with a specification.
444 :     \item The new \code{tmReduce()} transformation allows to combine multiple
445 :     maps into one transformation.
446 :     }
447 :     }
448 :     \subsection{DEPRECATED & DEFUNCT}{
449 :     \itemize{
450 :     \item CSVSource is defunct (use DataframeSource instead).
451 :     \item weightLogical is defunct.
452 :     \item TermDocMatrix is defunct (use DocumentTermMatrix or
453 :     TermDocumentMatrix instead).
454 :     }
455 :     }
456 :     }
457 :     \section{Changes in tm version 0.3-3}{
458 :     \subsection{NEW FEATURES}{
459 :     \itemize{
460 :     \item The abstract Source class gets a default implementation for
461 :     the \code{stepNext()} method. It increments the position counter by
462 :     one, a reasonable value for most sources. For special purposes
463 :     custom methods can be created via overloading \code{stepNext()} of
464 :     the subclass.
465 :     \item New URISource class for a single document identified by a
466 :     Uniform Resource Identifier.
467 :     \item New DataframeSource for documents stored in a data frame. Each
468 :     row is interpreted as a single document.
469 :     }
470 :     }
471 :     \subsection{BUG FIXES}{
472 :     \itemize{
473 :     \item Fix off-by-one error in \code{convertMboxEml()} function. Reported by
474 :     Angela Bohn.
475 :     \item Sort row indices in sparse term-document matrices. Kudos to
476 :     Martin Mächler for his suggestions.
477 :     \item Sources and readers no longer evaluate calls in a non-standard
478 :     way.
479 :     }
480 :     }
481 :     }
482 :     \section{Changes in tm version 0.3-2}{
483 :     \subsection{NEW FEATURES}{
484 :     \itemize{
485 :     \item Weighting functions now have an Acronym slot containing
486 :     abbreviations of the weighting functions' names. This is highly
487 :     useful when generating tables with indications which weighting
488 :     scheme was actually used for your experiments.
489 :     \item The functions \code{tmFilter()}, \code{tmIndex()}, \code{tmMap()} and \code{TermDocMatrix()}
490 : feinerer 1233 now can use a MPI cluster (via the \pkg{snow} and \pkg{Rmpi} packages) if
491 : feinerer 1170 available. Use \code{(de)activateCluster()} to manually override
492 :     cluster usage settings. Special thanks to Stefan Theussl for
493 :     his constructive comments.
494 :     \item The Source class receives a new Length slot. It contains the
495 :     number of elements provided by the source (although there
496 :     might be rare cases where the number cannot be determined in
497 :     advance---then it should be set to zero).
498 :     }
499 :     }
500 :     }

R-Forge@R-project.org
ViewVC Help
Powered by ViewVC 1.0.0  
Thanks to:
Vienna University of Economics and Business University of Wisconsin - Madison Powered By FusionForge