SCM

SCM Repository

[tm] Annotation of /pkg/inst/NEWS.Rd
ViewVC logotype

Annotation of /pkg/inst/NEWS.Rd

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1481 - (view) (download) (as text)

1 : feinerer 1170 \name{NEWS}
2 :     \title{News for Package 'tm'}
3 : khornik 1396 \encoding{UTF-8}
4 : feinerer 1474 \section{Changes in tm version 0.7-2}{
5 : feinerer 1481 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
6 :     \itemize{
7 :     \item \code{DataframeSource} now only processes data frames with the two
8 :     mandatory columns \code{"doc_id"} and \code{"text"}. Additional columns
9 :     are used as document level metadata. This implements compatibility with
10 :     \emph{Text Interchange Formats} corpora
11 :     (\url{https://github.com/ropensci/tif}).
12 :     \item \code{readTabular()} has been removed. Use \code{DataframeSource}
13 :     instead.
14 :     }
15 :     }
16 : feinerer 1474 \subsection{BUG FIXES}{
17 :     \itemize{
18 :     \item Correctly handle the \code{dictionary} argument when constructing a
19 : feinerer 1478 term-document matrix from a \code{SimpleCorpus} (reported by Joe
20 :     Corrigan) or from a \code{VCorpus} (reported by Mark Rosenstein).
21 : feinerer 1474 }
22 :     }
23 :     }
24 : khornik 1472 \section{Changes in tm version 0.7-1}{
25 :     \subsection{BUG FIXES}{
26 :     \itemize{
27 :     \item Compilation fixes for Clang's libc++.
28 :     }
29 :     }
30 :     }
31 : feinerer 1437 \section{Changes in tm version 0.7}{
32 : feinerer 1436 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
33 :     \itemize{
34 :     \item \code{inspect.TermDocumentMatrix()} now displays a sample instead
35 :     of the full matrix. The full dense representation is available via
36 :     \code{as.matrix()}.
37 :     }
38 :     }
39 :     \subsection{NEW FEATURES}{
40 :     \itemize{
41 : feinerer 1437 \item \code{SimpleCorpus} provides a corpus which is optimized for the
42 :     most common usage scenario: importing plain texts from files in a
43 : feinerer 1440 directory or directly from a vector in \R, preprocessing and transforming
44 : feinerer 1437 the texts, and finally exporting them to a term-document matrix. The aim
45 :     is to boost performance and minimize memory pressure. It loads all
46 :     documents into memory, and is designed for medium-sized to large data
47 :     sets.
48 : feinerer 1436 \item \code{inspect()} on text documents as a shorthand for
49 :     \code{writeLines(as.character())}.
50 : feinerer 1450 \item \code{findMostFreqTerms()} finds most frequent terms in a
51 :     document-term or term-document matrix, or a vector of term frequencies.
52 : feinerer 1466 \item \code{tm_parLapply()} is now internally used for the parallelization
53 :     of transformations, filters, and term-document matrix construction. The
54 :     preferred parallelization engine can be registered via
55 :     \code{tm_parLapply_engine()}. The default is to use no parallelization
56 :     (instead of \code{\link[parallel]{mclapply}} (package \pkg{parallel}) in
57 :     previous versions).
58 : feinerer 1436 }
59 :     }
60 :     }
61 : feinerer 1432 \section{Changes in tm version 0.6-2}{
62 :     \subsection{BUG FIXES}{
63 :     \itemize{
64 :     \item \code{format.PlainTextDocument()} now reports only one character
65 :     count for a whole document.
66 :     }
67 :     }
68 :     }
69 : feinerer 1397 \section{Changes in tm version 0.6-1}{
70 : feinerer 1432 \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
71 :     \itemize{
72 :     \item \code{format.PlainTextDocument()} now displays a compact
73 :     representation instead of the content. Use \code{as.character()} to
74 :     obtain the character content (which in turn can be applied to a corpus
75 :     via \code{lapply()}).
76 :     }
77 :     }
78 : feinerer 1397 \subsection{NEW FEATURES}{
79 :     \itemize{
80 : feinerer 1425 \item \code{ZipSource()} for processing ZIP files.
81 : feinerer 1397 \item Sources now provide \code{open()} and \code{close()}.
82 : feinerer 1425 \item \code{termFreq()} now accepts \code{Span_Tokenizer} and
83 :     \code{Token_Tokenizer} (both from package \pkg{NLP}) objects as
84 :     tokenizers.
85 :     \item \code{readTagged()}, a reader for text documents containing
86 :     POS-tagged words.
87 : feinerer 1397 }
88 :     }
89 : feinerer 1413 \subsection{BUG FIXES}{
90 :     \itemize{
91 :     \item The function \code{removeWords()} now correctly processes words
92 :     being truncations of others. Reported by Александр Труфанов.
93 :     }
94 :     }
95 : feinerer 1397 }
96 : feinerer 1345 \section{Changes in tm version 0.6}{
97 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
98 :     \itemize{
99 : feinerer 1368 \item \code{DirSource()} and \code{URISource()} now use the argument
100 :     \code{encoding} for conversion via \code{iconv()} to \code{"UTF-8"}.
101 :     \item \code{termFreq()} now uses \code{words()} as the default tokenizer.
102 : feinerer 1345 \item Text documents now provide the functions \code{content()} and
103 :     \code{as.character()} to access the (possibly raw) document content and
104 :     the natural language text in a suitable (not necessarily structured)
105 :     form.
106 : feinerer 1372 \item The internal representation of corpora, sources, and text documents
107 :     changed. Saved objects created with older \pkg{tm} versions are
108 :     incompatible and need to be rebuilt.
109 : feinerer 1345 }
110 :     }
111 :     \subsection{NEW FEATURES}{
112 :     \itemize{
113 :     \item \code{DirSource()} and \code{URISource()} now have a \code{mode}
114 :     argument specifying how elements should be read (no read, binary, text).
115 :     \item Improved high-level documentation on corpora (\code{?Corpus}), text
116 :     documents (\code{?TextDocument}), sources (\code{?Source}), and readers
117 :     (\code{?Reader}).
118 : feinerer 1368 \item Integration with package \pkg{NLP}.
119 :     \item Romanian stopwords. Suggested by Cristian Chirita.
120 :     \item \code{words.PlainTextDocument()} delivers word tokens in the
121 :     document.
122 : feinerer 1345 }
123 :     }
124 :     \subsection{BUG FIXES}{
125 :     \itemize{
126 :     \item The function \code{stemCompletion()} now avoids spurious duplicate
127 :     results. Reported by Seong-Hyeon Kim.
128 :     }
129 :     }
130 :     \subsection{DEPRECATED & DEFUNCT}{
131 :     \itemize{
132 :     \item Following functions have been removed:
133 :     \itemize{
134 :     \item \code{Author()}, \code{DateTimeStamp()}, \code{CMetaData()},
135 :     \code{content_meta()}, \code{DMetaData()}, \code{Description()},
136 :     \code{Heading()}, \code{ID()}, \code{Language()},
137 :     \code{LocalMetaData()}, \code{Origin()}, \code{prescindMeta()},
138 :     \code{sFilter()} (use \code{meta()} instead).
139 :     \item \code{dissimilarity()} (use \code{proxy::dist()} instead).
140 :     \item \code{makeChunks()} (use \code{[} and \code{[[} manually).
141 :     \item \code{summary.Corpus()} and \code{summary.TextRepository()}
142 :     (\code{print()} now gives a more informative but succinct overview).
143 :     \item \code{TextRepository()} and \code{RepoMetaData()} (use e.g. a
144 :     list to store multiple corpora instead).
145 :     }
146 :     }
147 :     }
148 :     }
149 : feinerer 1239 \section{Changes in tm version 0.5-10}{
150 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
151 :     \itemize{
152 :     \item License changed to GPL-3 (from GPL-2 | GPL-3).
153 : feinerer 1255 \item Following functions have been renamed:
154 :     \itemize{
155 : feinerer 1432 \item \code{tm_tag_score()} to \code{tm_term_score()}.
156 : feinerer 1255 }
157 : feinerer 1239 }
158 :     }
159 : feinerer 1242 \subsection{DEPRECATED & DEFUNCT}{
160 :     \itemize{
161 : feinerer 1253 \item Following functions have been removed:
162 :     \itemize{
163 : feinerer 1277 \item \code{Dictionary()} (use a character vector instead; use
164 : feinerer 1280 \code{Terms()} to extract terms from a document-term or term-document
165 : feinerer 1277 matrix),
166 : feinerer 1258 \item \code{GmaneSource()} (but still available via an example in
167 : feinerer 1283 \code{XMLSource()}),
168 : feinerer 1260 \item \code{preprocessReut21578XML()} (moved to package
169 :     \pkg{tm.corpus.Reuters21578}),
170 : feinerer 1258 \item \code{readGmane()} (but still available via an example in
171 : feinerer 1283 \code{readXML()}),
172 : feinerer 1432 \item \code{searchFullText()} and \code{tm_intersect()}
173 : feinerer 1253 (use \code{grep()} instead).
174 :     }
175 : feinerer 1242 \item Following S3 classes are no longer registered as S4 classes:
176 :     \itemize{
177 : feinerer 1432 \item \code{VCorpus} and \code{PlainTextDocument}.
178 : feinerer 1242 }
179 :     }
180 :     }
181 : feinerer 1239 }
182 : feinerer 1224 \section{Changes in tm version 0.5-9}{
183 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
184 :     \itemize{
185 : feinerer 1226 \item Stemming functionality is now provided by the package
186 :     \pkg{SnowballC} replacing packages \pkg{Snowball} and \pkg{RWeka}.
187 : feinerer 1224 \item All stopword lists (besides Catalan and SMART) available via
188 :     \code{stopwords()} now come from the Snowball stemmer project.
189 : feinerer 1227 \item Transformations, filters, and term-document matrix construction
190 :     now use \code{\link[parallel]{mclapply}} (package \pkg{parallel}).
191 :     Packages \pkg{snow} and \pkg{Rmpi} are no longer used.
192 : feinerer 1224 }
193 :     }
194 : feinerer 1227 \subsection{DEPRECATED & DEFUNCT}{
195 :     \itemize{
196 :     \item Following functions have been removed:
197 :     \itemize{
198 : feinerer 1432 \item \code{tm_startCluster()} and \code{tm_stopCluster()}.
199 : feinerer 1227 }
200 :     }
201 :     }
202 : feinerer 1224 }
203 : feinerer 1173 \section{Changes in tm version 0.5-8}{
204 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
205 :     \itemize{
206 :     \item The function \code{termFreq()} now processes the
207 :     \code{tolower} and \code{tokenize} options first.
208 :     }
209 :     }
210 : feinerer 1174 \subsection{NEW FEATURES}{
211 :     \itemize{
212 :     \item Catalan stopwords. Requested by Xavier Fernández i Marín.
213 :     }
214 :     }
215 : feinerer 1173 \subsection{BUG FIXES}{
216 :     \itemize{
217 :     \item The function \code{termFreq()} now correctly accepts
218 :     user-provided stopwords. Reported by Bettina Grün.
219 :     \item The function \code{termFreq()} now correctly handles the
220 :     lower bound of the option \code{wordLength}. Reported by Steven
221 :     C. Bagley.
222 :     }
223 :     }
224 :     }
225 : feinerer 1170 \section{Changes in tm version 0.5-7}{
226 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
227 :     \itemize{
228 :     \item The function \code{termFreq()} provides two new arguments for
229 :     generalized bounds checking of term frequencies and word
230 :     lengths. This replaces the arguments minDocFreq and
231 :     minWordLength.
232 :     \item The function \code{termFreq()} is now sensitive to the order of
233 :     control options.
234 :     }
235 :     }
236 :     \subsection{NEW FEATURES}{
237 :     \itemize{
238 :     \item Weighting schemata for term-document matrices in SMART notation.
239 :     \item Local and global options for term-document matrix
240 :     construction.
241 :     \item SMART stopword list was added.
242 :     }
243 :     }
244 :     }
245 :     \section{Changes in tm version 0.5-5}{
246 :     \subsection{NEW FEATURES}{
247 :     \itemize{
248 :     \item Access documents in a corpus by names (fallback to IDs if names are
249 :     not set), i.e., allow a string for the corpus operator `[[`.
250 :     }
251 :     }
252 :     \subsection{BUG FIXES}{
253 :     \itemize{
254 :     \item The function \code{findFreqTerms()} now checks bounds on a global level
255 :     (to comply with the manual page) instead per document. Reported
256 :     and fixed by Thomas Zapf-Schramm.
257 :     }
258 :     }
259 :     }
260 :     \section{Changes in tm version 0.5-4}{
261 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
262 :     \itemize{
263 :     \item Use IETF language tags for language codes (instead of ISO 639-2).
264 :     }
265 :     }
266 :     \subsection{NEW FEATURES}{
267 :     \itemize{
268 :     \item The function \code{tm_tag_score()} provides functionality to score
269 :     documents based on the number of tags found. This is useful for
270 :     sentiment analysis.
271 :     \item The weighting function for term frequency-inverse document
272 :     frequency \code{weightTfIdf()} has now an option for term
273 :     normalization.
274 :     \item Plotting functions to test for Zipf's and Heaps' law on a
275 :     term-document matrix were added: \code{Zipf_plot()} and
276 :     \code{Heaps_plot()}. Contributed by Kurt Hornik.
277 :     }
278 :     }
279 :     }
280 :     \section{Changes in tm version 0.5-3}{
281 :     \subsection{NEW FEATURES}{
282 :     \itemize{
283 :     \item The reader function \code{readRCV1asPlain()} was added and combines the
284 :     functionality of \code{readRCV1()} and \code{as.PlainTextDocument()}.
285 :     \item The function \code{stemCompletion()} has a set of new heuristics.
286 :     }
287 :     }
288 :     }
289 :     \section{Changes in tm version 0.5-2}{
290 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
291 :     \itemize{
292 :     \item The function \code{termFreq()} which is used for building a
293 :     term-document matrix now uses a whitespace oriented tokenizer
294 :     as default.
295 :     }
296 :     }
297 :     \subsection{NEW FEATURES}{
298 :     \itemize{
299 :     \item A combine method for merging multiple term-document matrices
300 :     was added (\code{c.TermDocumentMatrix()}).
301 :     \item The function \code{termFreq()} has now an option to remove
302 :     punctuation characters.
303 :     }
304 :     }
305 :     \subsection{DEPRECATED & DEFUNCT}{
306 :     \itemize{
307 :     \item Following functions have been removed:
308 :     \itemize{
309 : feinerer 1432 \item \code{CSVSource()} (use \code{DataframeSource(read.csv(..., stringsAsFactors = FALSE))} instead), and
310 : feinerer 1170 \item \code{TermDocMatrix()} (use \code{DocumentTermMatrix()} instead).
311 :     }
312 :     }
313 :     }
314 :     \subsection{BUG FIXES}{
315 :     \itemize{
316 :     \item \code{removeWords()} no longer skips words at the beginning or the end
317 :     of a line. Reported by Mark Kimpel.
318 :     }
319 :     }
320 :     }
321 :     \section{Changes in tm version 0.5-1}{
322 :     \subsection{BUG FIXES}{
323 :     \itemize{
324 :     \item \code{preprocessReut21578XML()} no longer generates invalid file names.
325 :     }
326 :     }
327 :     }
328 :     \section{Changes in tm version 0.5}{
329 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
330 :     \itemize{
331 :     \item All classes, functions, and generics are reimplemented using
332 :     the S3 class system.
333 :     \item Following functions have been renamed:
334 :     \itemize{
335 : feinerer 1432 \item \code{activateCluster()} to \code{tm_startCluster()},
336 : feinerer 1170 \item \code{asPlain()} to \code{as.PlainTextDocument()},
337 :     \item \code{deactivateCluster()} to \code{tm_stopCluster()},
338 :     \item \code{tmFilter()} to \code{tm_filter()},
339 :     \item \code{tmIndex()} to \code{tm_index()},
340 :     \item \code{tmIntersect()} to \code{tm_intersect()}, and
341 :     \item \code{tmMap()} to \code{tm_map()}.
342 :     }
343 :     \item Mail handling functionality is factored out to the
344 : feinerer 1233 \pkg{tm.plugin.mail} package.
345 : feinerer 1170 }
346 :     }
347 :     \subsection{DEPRECATED & DEFUNCT}{
348 :     \itemize{
349 :     \item Following functions have been removed:
350 :     \itemize{
351 :     \item \code{tmTolower()} (use \code{tolower()} instead), and
352 : feinerer 1172 \item \code{replacePatterns()} (use \code{gsub()} instead).
353 : feinerer 1170 }
354 :     }
355 :     }
356 :     }
357 :     \section{Changes in tm version 0.4}{
358 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
359 :     \itemize{
360 :     \item The Corpus class is now virtual providing an abstract
361 :     interface.
362 :     \item VCorpus, the default implementation of the abstract corpus
363 :     interface (by subclassing), provides a corpus with volatile (=
364 : feinerer 1438 standard \R object) semantics. It loads all documents into
365 : feinerer 1170 memory, and is designed for small to medium-sized data sets.
366 :     \item PCorpus, an implementation of the abstract corpus interface (by
367 :     subclassing), provides a corpus with permanent storage
368 :     semantics. The actual data is stored in an external database
369 : feinerer 1233 (file) object (as supported by the \pkg{filehash} package), with
370 : feinerer 1170 automatic (un-)loading into memory. It is designed for systems
371 :     with small memory.
372 :     \item Language codes are now in ISO 639-2 (instead of ISO 639-1).
373 :     \item Reader functions no longer have a load argument for lazy
374 :     loading.
375 :     }
376 :     }
377 :     \subsection{NEW FEATURES}{
378 :     \itemize{
379 :     \item The reader function \code{readReut21578XMLasPlain()} was added and
380 :     combines the functionality of \code{readReut21578XML()} and \code{asPlain()}.
381 :     }
382 :     }
383 :     \subsection{BUG FIXES}{
384 :     \itemize{
385 :     \item \code{weightTfIdf()} no longer applies a binary weighting to an input
386 :     matrix in term frequency format (which happened only in 0.3-4).
387 :     }
388 :     }
389 :     }
390 :     \section{Changes in tm version 0.3-4}{
391 :     \subsection{SIGNIFICANT USER-VISIBLE CHANGES}{
392 :     \itemize{
393 :     \item \code{.onLoad()} no longer tries to start a MPI cluster (which often
394 :     failed in misconfigured environments). Use \code{activateCluster()}
395 :     and \code{deactivateCluster()} instead.
396 :     \item DocumentTermMatrix (the improved reimplementation of defunct
397 : feinerer 1233 TermDocMatrix) does not use the \pkg{Matrix} package anymore.
398 : feinerer 1170 }
399 :     }
400 :     \subsection{NEW FEATURES}{
401 :     \itemize{
402 :     \item The \code{DirSource()} constructor now accepts the two new (optional)
403 :     arguments pattern and ignore.case. With pattern one can define
404 :     a regular expression for selecting only matching files, and
405 :     ignore.case specifies whether pattern-matching is
406 :     case-sensitive.
407 :     \item The \code{readNewsgroup()} reader function can now be configured for
408 :     custom date formats (via the DateFormat argument).
409 :     \item The \code{readPDF()} reader function can now be configured (via the
410 :     PdfinfoOptions and PdftotextOptions arguments).
411 :     \item The \code{readDOC()} reader function can now be configured (via the
412 :     AntiwordOptions argument).
413 :     \item Sources now can be vectorized. This allows faster corpus
414 :     construction.
415 :     \item New XMLSource class for arbitrary XML files.
416 :     \item The new \code{readTabular()} reader function allows to create a custom
417 :     tailor-made reader configured via mappings from a tabular data
418 :     structure.
419 :     \item The new \code{readXML()} reader function allows to read in arbitrary
420 :     XML files which are described with a specification.
421 :     \item The new \code{tmReduce()} transformation allows to combine multiple
422 :     maps into one transformation.
423 :     }
424 :     }
425 :     \subsection{DEPRECATED & DEFUNCT}{
426 :     \itemize{
427 :     \item CSVSource is defunct (use DataframeSource instead).
428 :     \item weightLogical is defunct.
429 :     \item TermDocMatrix is defunct (use DocumentTermMatrix or
430 :     TermDocumentMatrix instead).
431 :     }
432 :     }
433 :     }
434 :     \section{Changes in tm version 0.3-3}{
435 :     \subsection{NEW FEATURES}{
436 :     \itemize{
437 :     \item The abstract Source class gets a default implementation for
438 :     the \code{stepNext()} method. It increments the position counter by
439 :     one, a reasonable value for most sources. For special purposes
440 :     custom methods can be created via overloading \code{stepNext()} of
441 :     the subclass.
442 :     \item New URISource class for a single document identified by a
443 :     Uniform Resource Identifier.
444 :     \item New DataframeSource for documents stored in a data frame. Each
445 :     row is interpreted as a single document.
446 :     }
447 :     }
448 :     \subsection{BUG FIXES}{
449 :     \itemize{
450 :     \item Fix off-by-one error in \code{convertMboxEml()} function. Reported by
451 :     Angela Bohn.
452 :     \item Sort row indices in sparse term-document matrices. Kudos to
453 :     Martin Mächler for his suggestions.
454 :     \item Sources and readers no longer evaluate calls in a non-standard
455 :     way.
456 :     }
457 :     }
458 :     }
459 :     \section{Changes in tm version 0.3-2}{
460 :     \subsection{NEW FEATURES}{
461 :     \itemize{
462 :     \item Weighting functions now have an Acronym slot containing
463 :     abbreviations of the weighting functions' names. This is highly
464 :     useful when generating tables with indications which weighting
465 :     scheme was actually used for your experiments.
466 :     \item The functions \code{tmFilter()}, \code{tmIndex()}, \code{tmMap()} and \code{TermDocMatrix()}
467 : feinerer 1233 now can use a MPI cluster (via the \pkg{snow} and \pkg{Rmpi} packages) if
468 : feinerer 1170 available. Use \code{(de)activateCluster()} to manually override
469 :     cluster usage settings. Special thanks to Stefan Theussl for
470 :     his constructive comments.
471 :     \item The Source class receives a new Length slot. It contains the
472 :     number of elements provided by the source (although there
473 :     might be rare cases where the number cannot be determined in
474 :     advance---then it should be set to zero).
475 :     }
476 :     }
477 :     }

R-Forge@R-project.org
ViewVC Help
Powered by ViewVC 1.0.0  
Thanks to:
Vienna University of Economics and Business University of Wisconsin - Madison Powered By FusionForge