[#6625] Global Parameter For VM Allocation To Prevent "Error: cannot allocate vector of size x Gb"

2019-04-29 16:34
Submitted by:
JAMES Bowery (jabowery)
Assigned to:
Nobody (None)
Operating System:
Global Parameter For VM Allocation To Prevent "Error: cannot allocate vector of size x Gb"

Detailed description
When using packages that, in turn, use the Matrix package, it is often difficult to estimate the maximum RAM allocation. This results in possibly days of computation being discarded with the error:

Error: cannot allocate vector of size x Gb

With the understanding that VM can result in paging and corresponding dramatic slowdown, nevertheless a global parameter set at the application level to use VM rather than RAM allocation would avoid this kind of hard failure. The main benefit of this is that the computation can continue while other options for completing the computation successfully are being explored, such as starting the computation in the cloud on a machine with more RAM, without terminating the computation already in progress on the chance that it will complete before the higher capacity computation completes.


Message  ↓
Date: 2023-06-27 17:50
Sender: Mikael Jagan

Closing now, absent further information ...

Date: 2022-08-21 22:34
Sender: Mikael Jagan

Please provide a minimal example demonstrating the problem, preferably relying only on 'Matrix' and base R. Where feasible, we try to allocate up front so that such errors are thrown early. It's possible that we can do better in your use case, but it's hard to know without an example...

As for your specific proposal of a "global parameter", can you elaborate? I.e., can you provide some code suggesting an implementation?

Note that 'Matrix', much like base R, is designed to provide "basic" functionality, which others can extend, wrap, etc. as they please (see, e.g., 'MatrixExtra'). So, you might consider encapsulating your "feature" in a package of its own.

Attached Files:


Field Old Value Date By
close_dateNone2023-06-27 17:50jaganmn
status_idOpen2023-06-27 17:50jaganmn
Thanks to:
Vienna University of Economics and Business Powered By FusionForge