SCM

SCM Repository

[tm] Annotation of /pkg/inst/NEWS.Rd
ViewVC logotype

Annotation of /pkg/inst/NEWS.Rd

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1495 - (view) (download) (as text)

1 : feinerer 1170 \name{NEWS}
2 :     \title{News for Package 'tm'}
3 : khornik 1396 \encoding{UTF-8}
4 : feinerer 1474 \section{Changes in tm version 0.7-2}{
5 : feinerer 1481 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
6 :     \itemize{
7 :     \item \code{DataframeSource} now only processes data frames with the two
8 :     mandatory columns \code{"doc_id"} and \code{"text"}. Additional columns
9 :     are used as document level metadata. This implements compatibility with
10 :     \emph{Text Interchange Formats} corpora
11 :     (\url{https://github.com/ropensci/tif}).
12 :     \item \code{readTabular()} has been removed. Use \code{DataframeSource}
13 :     instead.
14 : feinerer 1495 \item \code{removeNumbers()} and \code{removePunctuation()} now have an
15 :     argument \code{ucp} to check for Unicode general categories \code{Nd}
16 :     (decimal digits) and \code{P} (punctuation), respectively. Contributed
17 :     by Kurt Hornik.
18 : feinerer 1481 }
19 :     }
20 : feinerer 1488 \subsection{NEW FEATURES}{
21 :     \itemize{
22 :     \item \code{Boost_tokenizer} provides a tokenizer based on the Boost
23 :     (\url{http://www.boost.org}) Tokenizer.
24 :     }
25 :     }
26 : feinerer 1474 \subsection{BUG FIXES}{
27 :     \itemize{
28 :     \item Correctly handle the \code{dictionary} argument when constructing a
29 : feinerer 1478 term-document matrix from a \code{SimpleCorpus} (reported by Joe
30 :     Corrigan) or from a \code{VCorpus} (reported by Mark Rosenstein).
31 : feinerer 1474 }
32 :     }
33 :     }
34 : khornik 1472 \section{Changes in tm version 0.7-1}{
35 :     \subsection{BUG FIXES}{
36 :     \itemize{
37 :     \item Compilation fixes for Clang's libc++.
38 :     }
39 :     }
40 :     }
41 : feinerer 1437 \section{Changes in tm version 0.7}{
42 : feinerer 1436 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
43 :     \itemize{
44 :     \item \code{inspect.TermDocumentMatrix()} now displays a sample instead
45 :     of the full matrix. The full dense representation is available via
46 :     \code{as.matrix()}.
47 :     }
48 :     }
49 :     \subsection{NEW FEATURES}{
50 :     \itemize{
51 : feinerer 1437 \item \code{SimpleCorpus} provides a corpus which is optimized for the
52 :     most common usage scenario: importing plain texts from files in a
53 : feinerer 1440 directory or directly from a vector in \R, preprocessing and transforming
54 : feinerer 1437 the texts, and finally exporting them to a term-document matrix. The aim
55 :     is to boost performance and minimize memory pressure. It loads all
56 :     documents into memory, and is designed for medium-sized to large data
57 :     sets.
58 : feinerer 1436 \item \code{inspect()} on text documents as a shorthand for
59 :     \code{writeLines(as.character())}.
60 : feinerer 1450 \item \code{findMostFreqTerms()} finds most frequent terms in a
61 :     document-term or term-document matrix, or a vector of term frequencies.
62 : feinerer 1466 \item \code{tm_parLapply()} is now internally used for the parallelization
63 :     of transformations, filters, and term-document matrix construction. The
64 :     preferred parallelization engine can be registered via
65 :     \code{tm_parLapply_engine()}. The default is to use no parallelization
66 :     (instead of \code{\link[parallel]{mclapply}} (package \pkg{parallel}) in
67 :     previous versions).
68 : feinerer 1436 }
69 :     }
70 :     }
71 : feinerer 1432 \section{Changes in tm version 0.6-2}{
72 :     \subsection{BUG FIXES}{
73 :     \itemize{
74 :     \item \code{format.PlainTextDocument()} now reports only one character
75 :     count for a whole document.
76 :     }
77 :     }
78 :     }
79 : feinerer 1397 \section{Changes in tm version 0.6-1}{
80 : feinerer 1432 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
81 :     \itemize{
82 :     \item \code{format.PlainTextDocument()} now displays a compact
83 :     representation instead of the content. Use \code{as.character()} to
84 :     obtain the character content (which in turn can be applied to a corpus
85 :     via \code{lapply()}).
86 :     }
87 :     }
88 : feinerer 1397 \subsection{NEW FEATURES}{
89 :     \itemize{
90 : feinerer 1425 \item \code{ZipSource()} for processing ZIP files.
91 : feinerer 1397 \item Sources now provide \code{open()} and \code{close()}.
92 : feinerer 1425 \item \code{termFreq()} now accepts \code{Span_Tokenizer} and
93 :     \code{Token_Tokenizer} (both from package \pkg{NLP}) objects as
94 :     tokenizers.
95 :     \item \code{readTagged()}, a reader for text documents containing
96 :     POS-tagged words.
97 : feinerer 1397 }
98 :     }
99 : feinerer 1413 \subsection{BUG FIXES}{
100 :     \itemize{
101 :     \item The function \code{removeWords()} now correctly processes words
102 :     being truncations of others. Reported by Александр Труфанов.
103 :     }
104 :     }
105 : feinerer 1397 }
106 : feinerer 1345 \section{Changes in tm version 0.6}{
107 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
108 :     \itemize{
109 : feinerer 1368 \item \code{DirSource()} and \code{URISource()} now use the argument
110 :     \code{encoding} for conversion via \code{iconv()} to \code{"UTF-8"}.
111 :     \item \code{termFreq()} now uses \code{words()} as the default tokenizer.
112 : feinerer 1345 \item Text documents now provide the functions \code{content()} and
113 :     \code{as.character()} to access the (possibly raw) document content and
114 :     the natural language text in a suitable (not necessarily structured)
115 :     form.
116 : feinerer 1372 \item The internal representation of corpora, sources, and text documents
117 :     changed. Saved objects created with older \pkg{tm} versions are
118 :     incompatible and need to be rebuilt.
119 : feinerer 1345 }
120 :     }
121 :     \subsection{NEW FEATURES}{
122 :     \itemize{
123 :     \item \code{DirSource()} and \code{URISource()} now have a \code{mode}
124 :     argument specifying how elements should be read (no read, binary, text).
125 :     \item Improved high-level documentation on corpora (\code{?Corpus}), text
126 :     documents (\code{?TextDocument}), sources (\code{?Source}), and readers
127 :     (\code{?Reader}).
128 : feinerer 1368 \item Integration with package \pkg{NLP}.
129 :     \item Romanian stopwords. Suggested by Cristian Chirita.
130 :     \item \code{words.PlainTextDocument()} delivers word tokens in the
131 :     document.
132 : feinerer 1345 }
133 :     }
134 :     \subsection{BUG FIXES}{
135 :     \itemize{
136 :     \item The function \code{stemCompletion()} now avoids spurious duplicate
137 :     results. Reported by Seong-Hyeon Kim.
138 :     }
139 :     }
140 :     \subsection{DEPRECATED & DEFUNCT}{
141 :     \itemize{
142 :     \item Following functions have been removed:
143 :     \itemize{
144 :     \item \code{Author()}, \code{DateTimeStamp()}, \code{CMetaData()},
145 :     \code{content_meta()}, \code{DMetaData()}, \code{Description()},
146 :     \code{Heading()}, \code{ID()}, \code{Language()},
147 :     \code{LocalMetaData()}, \code{Origin()}, \code{prescindMeta()},
148 :     \code{sFilter()} (use \code{meta()} instead).
149 :     \item \code{dissimilarity()} (use \code{proxy::dist()} instead).
150 :     \item \code{makeChunks()} (use \code{[} and \code{[[} manually).
151 :     \item \code{summary.Corpus()} and \code{summary.TextRepository()}
152 :     (\code{print()} now gives a more informative but succinct overview).
153 :     \item \code{TextRepository()} and \code{RepoMetaData()} (use e.g. a
154 :     list to store multiple corpora instead).
155 :     }
156 :     }
157 :     }
158 :     }
159 : feinerer 1239 \section{Changes in tm version 0.5-10}{
160 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
161 :     \itemize{
162 :     \item License changed to GPL-3 (from GPL-2 | GPL-3).
163 : feinerer 1255 \item Following functions have been renamed:
164 :     \itemize{
165 : feinerer 1432 \item \code{tm_tag_score()} to \code{tm_term_score()}.
166 : feinerer 1255 }
167 : feinerer 1239 }
168 :     }
169 : feinerer 1242 \subsection{DEPRECATED & DEFUNCT}{
170 :     \itemize{
171 : feinerer 1253 \item Following functions have been removed:
172 :     \itemize{
173 : feinerer 1277 \item \code{Dictionary()} (use a character vector instead; use
174 : feinerer 1280 \code{Terms()} to extract terms from a document-term or term-document
175 : feinerer 1277 matrix),
176 : feinerer 1258 \item \code{GmaneSource()} (but still available via an example in
177 : feinerer 1283 \code{XMLSource()}),
178 : feinerer 1260 \item \code{preprocessReut21578XML()} (moved to package
179 :     \pkg{tm.corpus.Reuters21578}),
180 : feinerer 1258 \item \code{readGmane()} (but still available via an example in
181 : feinerer 1283 \code{readXML()}),
182 : feinerer 1432 \item \code{searchFullText()} and \code{tm_intersect()}
183 : feinerer 1253 (use \code{grep()} instead).
184 :     }
185 : feinerer 1242 \item Following S3 classes are no longer registered as S4 classes:
186 :     \itemize{
187 : feinerer 1432 \item \code{VCorpus} and \code{PlainTextDocument}.
188 : feinerer 1242 }
189 :     }
190 :     }
191 : feinerer 1239 }
192 : feinerer 1224 \section{Changes in tm version 0.5-9}{
193 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
194 :     \itemize{
195 : feinerer 1226 \item Stemming functionality is now provided by the package
196 :     \pkg{SnowballC} replacing packages \pkg{Snowball} and \pkg{RWeka}.
197 : feinerer 1224 \item All stopword lists (besides Catalan and SMART) available via
198 :     \code{stopwords()} now come from the Snowball stemmer project.
199 : feinerer 1227 \item Transformations, filters, and term-document matrix construction
200 :     now use \code{\link[parallel]{mclapply}} (package \pkg{parallel}).
201 :     Packages \pkg{snow} and \pkg{Rmpi} are no longer used.
202 : feinerer 1224 }
203 :     }
204 : feinerer 1227 \subsection{DEPRECATED & DEFUNCT}{
205 :     \itemize{
206 :     \item Following functions have been removed:
207 :     \itemize{
208 : feinerer 1432 \item \code{tm_startCluster()} and \code{tm_stopCluster()}.
209 : feinerer 1227 }
210 :     }
211 :     }
212 : feinerer 1224 }
213 : feinerer 1173 \section{Changes in tm version 0.5-8}{
214 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
215 :     \itemize{
216 :     \item The function \code{termFreq()} now processes the
217 :     \code{tolower} and \code{tokenize} options first.
218 :     }
219 :     }
220 : feinerer 1174 \subsection{NEW FEATURES}{
221 :     \itemize{
222 :     \item Catalan stopwords. Requested by Xavier Fernández i Marín.
223 :     }
224 :     }
225 : feinerer 1173 \subsection{BUG FIXES}{
226 :     \itemize{
227 :     \item The function \code{termFreq()} now correctly accepts
228 :     user-provided stopwords. Reported by Bettina Grün.
229 :     \item The function \code{termFreq()} now correctly handles the
230 :     lower bound of the option \code{wordLength}. Reported by Steven
231 :     C. Bagley.
232 :     }
233 :     }
234 :     }
235 : feinerer 1170 \section{Changes in tm version 0.5-7}{
236 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
237 :     \itemize{
238 :     \item The function \code{termFreq()} provides two new arguments for
239 :     generalized bounds checking of term frequencies and word
240 :     lengths. This replaces the arguments minDocFreq and
241 :     minWordLength.
242 :     \item The function \code{termFreq()} is now sensitive to the order of
243 :     control options.
244 :     }
245 :     }
246 :     \subsection{NEW FEATURES}{
247 :     \itemize{
248 :     \item Weighting schemata for term-document matrices in SMART notation.
249 :     \item Local and global options for term-document matrix
250 :     construction.
251 :     \item SMART stopword list was added.
252 :     }
253 :     }
254 :     }
255 :     \section{Changes in tm version 0.5-5}{
256 :     \subsection{NEW FEATURES}{
257 :     \itemize{
258 :     \item Access documents in a corpus by names (fallback to IDs if names are
259 :     not set), i.e., allow a string for the corpus operator `[[`.
260 :     }
261 :     }
262 :     \subsection{BUG FIXES}{
263 :     \itemize{
264 :     \item The function \code{findFreqTerms()} now checks bounds on a global level
265 :     (to comply with the manual page) instead per document. Reported
266 :     and fixed by Thomas Zapf-Schramm.
267 :     }
268 :     }
269 :     }
270 :     \section{Changes in tm version 0.5-4}{
271 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
272 :     \itemize{
273 :     \item Use IETF language tags for language codes (instead of ISO 639-2).
274 :     }
275 :     }
276 :     \subsection{NEW FEATURES}{
277 :     \itemize{
278 :     \item The function \code{tm_tag_score()} provides functionality to score
279 :     documents based on the number of tags found. This is useful for
280 :     sentiment analysis.
281 :     \item The weighting function for term frequency-inverse document
282 :     frequency \code{weightTfIdf()} has now an option for term
283 :     normalization.
284 :     \item Plotting functions to test for Zipf's and Heaps' law on a
285 :     term-document matrix were added: \code{Zipf_plot()} and
286 :     \code{Heaps_plot()}. Contributed by Kurt Hornik.
287 :     }
288 :     }
289 :     }
290 :     \section{Changes in tm version 0.5-3}{
291 :     \subsection{NEW FEATURES}{
292 :     \itemize{
293 :     \item The reader function \code{readRCV1asPlain()} was added and combines the
294 :     functionality of \code{readRCV1()} and \code{as.PlainTextDocument()}.
295 :     \item The function \code{stemCompletion()} has a set of new heuristics.
296 :     }
297 :     }
298 :     }
299 :     \section{Changes in tm version 0.5-2}{
300 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
301 :     \itemize{
302 :     \item The function \code{termFreq()} which is used for building a
303 :     term-document matrix now uses a whitespace oriented tokenizer
304 :     as default.
305 :     }
306 :     }
307 :     \subsection{NEW FEATURES}{
308 :     \itemize{
309 :     \item A combine method for merging multiple term-document matrices
310 :     was added (\code{c.TermDocumentMatrix()}).
311 :     \item The function \code{termFreq()} has now an option to remove
312 :     punctuation characters.
313 :     }
314 :     }
315 :     \subsection{DEPRECATED & DEFUNCT}{
316 :     \itemize{
317 :     \item Following functions have been removed:
318 :     \itemize{
319 : feinerer 1432 \item \code{CSVSource()} (use \code{DataframeSource(read.csv(..., stringsAsFactors = FALSE))} instead), and
320 : feinerer 1170 \item \code{TermDocMatrix()} (use \code{DocumentTermMatrix()} instead).
321 :     }
322 :     }
323 :     }
324 :     \subsection{BUG FIXES}{
325 :     \itemize{
326 :     \item \code{removeWords()} no longer skips words at the beginning or the end
327 :     of a line. Reported by Mark Kimpel.
328 :     }
329 :     }
330 :     }
331 :     \section{Changes in tm version 0.5-1}{
332 :     \subsection{BUG FIXES}{
333 :     \itemize{
334 :     \item \code{preprocessReut21578XML()} no longer generates invalid file names.
335 :     }
336 :     }
337 :     }
338 :     \section{Changes in tm version 0.5}{
339 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
340 :     \itemize{
341 :     \item All classes, functions, and generics are reimplemented using
342 :     the S3 class system.
343 :     \item Following functions have been renamed:
344 :     \itemize{
345 : feinerer 1432 \item \code{activateCluster()} to \code{tm_startCluster()},
346 : feinerer 1170 \item \code{asPlain()} to \code{as.PlainTextDocument()},
347 :     \item \code{deactivateCluster()} to \code{tm_stopCluster()},
348 :     \item \code{tmFilter()} to \code{tm_filter()},
349 :     \item \code{tmIndex()} to \code{tm_index()},
350 :     \item \code{tmIntersect()} to \code{tm_intersect()}, and
351 :     \item \code{tmMap()} to \code{tm_map()}.
352 :     }
353 :     \item Mail handling functionality is factored out to the
354 : feinerer 1233 \pkg{tm.plugin.mail} package.
355 : feinerer 1170 }
356 :     }
357 :     \subsection{DEPRECATED & DEFUNCT}{
358 :     \itemize{
359 :     \item Following functions have been removed:
360 :     \itemize{
361 :     \item \code{tmTolower()} (use \code{tolower()} instead), and
362 : feinerer 1172 \item \code{replacePatterns()} (use \code{gsub()} instead).
363 : feinerer 1170 }
364 :     }
365 :     }
366 :     }
367 :     \section{Changes in tm version 0.4}{
368 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
369 :     \itemize{
370 :     \item The Corpus class is now virtual providing an abstract
371 :     interface.
372 :     \item VCorpus, the default implementation of the abstract corpus
373 :     interface (by subclassing), provides a corpus with volatile (=
374 : feinerer 1438 standard \R object) semantics. It loads all documents into
375 : feinerer 1170 memory, and is designed for small to medium-sized data sets.
376 :     \item PCorpus, an implementation of the abstract corpus interface (by
377 :     subclassing), provides a corpus with permanent storage
378 :     semantics. The actual data is stored in an external database
379 : feinerer 1233 (file) object (as supported by the \pkg{filehash} package), with
380 : feinerer 1170 automatic (un-)loading into memory. It is designed for systems
381 :     with small memory.
382 :     \item Language codes are now in ISO 639-2 (instead of ISO 639-1).
383 :     \item Reader functions no longer have a load argument for lazy
384 :     loading.
385 :     }
386 :     }
387 :     \subsection{NEW FEATURES}{
388 :     \itemize{
389 :     \item The reader function \code{readReut21578XMLasPlain()} was added and
390 :     combines the functionality of \code{readReut21578XML()} and \code{asPlain()}.
391 :     }
392 :     }
393 :     \subsection{BUG FIXES}{
394 :     \itemize{
395 :     \item \code{weightTfIdf()} no longer applies a binary weighting to an input
396 :     matrix in term frequency format (which happened only in 0.3-4).
397 :     }
398 :     }
399 :     }
400 :     \section{Changes in tm version 0.3-4}{
401 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
402 :     \itemize{
403 :     \item \code{.onLoad()} no longer tries to start a MPI cluster (which often
404 :     failed in misconfigured environments). Use \code{activateCluster()}
405 :     and \code{deactivateCluster()} instead.
406 :     \item DocumentTermMatrix (the improved reimplementation of defunct
407 : feinerer 1233 TermDocMatrix) does not use the \pkg{Matrix} package anymore.
408 : feinerer 1170 }
409 :     }
410 :     \subsection{NEW FEATURES}{
411 :     \itemize{
412 :     \item The \code{DirSource()} constructor now accepts the two new (optional)
413 :     arguments pattern and ignore.case. With pattern one can define
414 :     a regular expression for selecting only matching files, and
415 :     ignore.case specifies whether pattern-matching is
416 :     case-sensitive.
417 :     \item The \code{readNewsgroup()} reader function can now be configured for
418 :     custom date formats (via the DateFormat argument).
419 :     \item The \code{readPDF()} reader function can now be configured (via the
420 :     PdfinfoOptions and PdftotextOptions arguments).
421 :     \item The \code{readDOC()} reader function can now be configured (via the
422 :     AntiwordOptions argument).
423 :     \item Sources now can be vectorized. This allows faster corpus
424 :     construction.
425 :     \item New XMLSource class for arbitrary XML files.
426 :     \item The new \code{readTabular()} reader function allows to create a custom
427 :     tailor-made reader configured via mappings from a tabular data
428 :     structure.
429 :     \item The new \code{readXML()} reader function allows to read in arbitrary
430 :     XML files which are described with a specification.
431 :     \item The new \code{tmReduce()} transformation allows to combine multiple
432 :     maps into one transformation.
433 :     }
434 :     }
435 :     \subsection{DEPRECATED & DEFUNCT}{
436 :     \itemize{
437 :     \item CSVSource is defunct (use DataframeSource instead).
438 :     \item weightLogical is defunct.
439 :     \item TermDocMatrix is defunct (use DocumentTermMatrix or
440 :     TermDocumentMatrix instead).
441 :     }
442 :     }
443 :     }
444 :     \section{Changes in tm version 0.3-3}{
445 :     \subsection{NEW FEATURES}{
446 :     \itemize{
447 :     \item The abstract Source class gets a default implementation for
448 :     the \code{stepNext()} method. It increments the position counter by
449 :     one, a reasonable value for most sources. For special purposes
450 :     custom methods can be created via overloading \code{stepNext()} of
451 :     the subclass.
452 :     \item New URISource class for a single document identified by a
453 :     Uniform Resource Identifier.
454 :     \item New DataframeSource for documents stored in a data frame. Each
455 :     row is interpreted as a single document.
456 :     }
457 :     }
458 :     \subsection{BUG FIXES}{
459 :     \itemize{
460 :     \item Fix off-by-one error in \code{convertMboxEml()} function. Reported by
461 :     Angela Bohn.
462 :     \item Sort row indices in sparse term-document matrices. Kudos to
463 :     Martin Mächler for his suggestions.
464 :     \item Sources and readers no longer evaluate calls in a non-standard
465 :     way.
466 :     }
467 :     }
468 :     }
469 :     \section{Changes in tm version 0.3-2}{
470 :     \subsection{NEW FEATURES}{
471 :     \itemize{
472 :     \item Weighting functions now have an Acronym slot containing
473 :     abbreviations of the weighting functions' names. This is highly
474 :     useful when generating tables with indications which weighting
475 :     scheme was actually used for your experiments.
476 :     \item The functions \code{tmFilter()}, \code{tmIndex()}, \code{tmMap()} and \code{TermDocMatrix()}
477 : feinerer 1233 now can use a MPI cluster (via the \pkg{snow} and \pkg{Rmpi} packages) if
478 : feinerer 1170 available. Use \code{(de)activateCluster()} to manually override
479 :     cluster usage settings. Special thanks to Stefan Theussl for
480 :     his constructive comments.
481 :     \item The Source class receives a new Length slot. It contains the
482 :     number of elements provided by the source (although there
483 :     might be rare cases where the number cannot be determined in
484 :     advance---then it should be set to zero).
485 :     }
486 :     }
487 :     }

R-Forge@R-project.org
ViewVC Help
Powered by ViewVC 1.0.0  
Thanks to:
Vienna University of Economics and Business University of Wisconsin - Madison Powered By FusionForge