Jump to content

RAMDisk v2 - Store files in memory with almost unlimited capacity!

24 replies to this topic

#21 Mr. Bateman

  • Members
  • 98 posts
  • LocationAustralia

Posted 23 July 2014 - 10:31 AM

I'm working on a new update, which helps it handle massive images and drives (>100kb, i'm guessing), but I seem to be running into a wall.
To copy a 200kb file to a ramdisk, it takes 98 seconds (1:38) and saving the same ramdisk creates an archive about 2.6MB, which is a massive 1250% of the original size.
The first problem (and by a minor extension second) would be easily fixed by saving each file's data as a single string instead of a byte array, but due to LuaJ not properly converting bytes to their character counterpart, files would be corrupted. The second problem could be fixed by not serializing the table to be able to save it, but I do not know of a method to do so. (If someone could direct me to a much better method it would be greatly appreciated.) It could also be fixed by using a compressor, but that uses much more time to process and when I used a compressor to check if it could be done, it really struggled to the point of failing.
I'm still working on a RAID implementation I discussed earlier, and I guess this is the beginning of the "Huge Data" era.

#22 Mr. Bateman

  • Members
  • 98 posts
  • LocationAustralia

Posted 27 July 2014 - 02:20 AM

I've been working on solutions, and I'll post my progress so far:
  • The time it takes to load from a saved archive is now reduced to (almost) nothing, by putting a 'return ' at the start of the archive and using loadfile() to give it instead of the bulky unserialize
  • Archives are now reduced to a 300% ratio instead of the 1250% by removing almost all whitespace, and instead of storing numbers in the archive, it now saves the hex counterpart for a tad more compression. (I doubt I can even get it down to 200%, as anything lower would require saving it in binary mode, which takes too much time)
  • I'm working on rewriting the virtual filesystem API so that it holds a hex string instead of a table of bytes for each file, which should decrease the archive size even more and make loading anything instantaneous by using this as a sidestep to LuaJ's >128 ASCII issue.
Just as a heads up, archives saved in version 1.0 will be incompatible with 2.0. Copy the files over instead. (Not ramdisk save.)

#23 Mr. Bateman

  • Members
  • 98 posts
  • LocationAustralia

Posted 14 August 2014 - 04:16 AM

Version 2 has been released, you can download it now via the same pastebin link.
It should tackle issues related to big files which was it's prime purpose. Keep an eye on this for a FTP implementation in the works.

#24 !!!!!!!!!!ExclaimationMark

  • Members
  • 30 posts

Posted 14 August 2014 - 07:42 AM

so it's like tmpfs on [CENSORED].

Edited by !!!!!!!!!!ExclaimationMark, 14 August 2014 - 07:42 AM.

#25 logsys

  • Members
  • 171 posts

Posted 10 October 2014 - 02:24 PM

Wow, such improvement.. although just had an idea now for a use fron that

1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users