Jump to content




BBPack - Pastebin uploader / downloader

utility

11 replies to this topic

#1 Bomb Bloke

    Hobbyist Coder

  • Moderators
  • 7,099 posts
  • LocationTasmania (AU)

Posted 29 January 2015 - 06:30 AM

pastebin get cUYTGbpb bbpack

(Formerly called "Package")

This script is basically a modified version of the fabled "pastebin" script - it can send or receive files to or from your computer.

However, it has a couple of differences. First off, it supports multiple files per paste - tell it to upload a directory, and it'll do just that. The whole lot can then be downloaded elsewhere with one command. Whole system backups made easy!

On top of compressing with LZW, it also translates the upload into something ASCII-compatible, meaning that ComputerCraft won't strip out certain symbols as it would if you were using the regular pastebin script. Although this means the results aren't easily human-readable, it allows transmission of data that would otherwise be difficult to obtain when playing on an online MineCraft server (such as NBS files for Note, or schematic files for this schematic builder).

Note that bbpack's uploads will usually trigger pastebin's spam filtering system. You'll hence likely need to look up your paste in a web browser and fill in a captcha to complete the uploading process.

As of v1.6.0, BBPack implements a file system wrapper when loaded. By default this only adds a RAM drive (accessible via the root of your drive as "ram"), though it can also be configured to compress all of your files transparently, mount online files as if they were on your local drive, and connect to "clusters" - other ComputerCraft systems acting as combined storage drives.

It can either be executed as a regular script, or loaded as an API:

Script Usage

API Usage

File names are included in each paste for all files, so if you eg upload a single file, it's up to you whether you download it under a new name (by specifying one with the "get" command) or not (by using the "get" command on its own).

For example, if you wanted to download this pre-packed set of NBS files (as included with Note Block Studio), you'd enter:

bbpack get CYRmLz78 Songs

Version History

Edited by Bomb Bloke, 07 September 2017 - 12:44 PM.


#2 Cranium

    Ninja Scripter

  • Moderators
  • 4,031 posts
  • LocationLincoln, Nebraska

Posted 29 January 2015 - 09:15 AM

Very nice! Very well written.

#3 Bomb Bloke

    Hobbyist Coder

  • Moderators
  • 7,099 posts
  • LocationTasmania (AU)

Posted 03 February 2015 - 10:20 AM

Thanks. :)

I've updated the paste with a version that can also be loaded as an API. This makes it easier for other scripts to work with compression or base64 conversions - handy for eking extra disk space out of computers, or for transmitting odd files over rednet and so on.

#4 TheOddByte

    Lazy Coder

  • Members
  • 1,607 posts
  • LocationSweden

Posted 03 February 2015 - 01:14 PM

This is something that will be very handy, it'll be really useful for installers. And nice to see that you've updated it.
I'll certainly be using this sometime in the future :D

#5 Bomb Bloke

    Hobbyist Coder

  • Moderators
  • 7,099 posts
  • LocationTasmania (AU)

Posted 17 April 2015 - 11:17 AM

Made a few tweaks, such as adding a package.open() function to the API side of things. It returns a handle that's pretty much the same as what you'd get out of fs.open(), but rigged to pass any data you hand it through its compression system. Which you could already do yourself using the existing functions, albeit without the convenience.

Eg, just as you can do this:

local myFile = fs.open("somefile", "w")
myFile.writeLine("some text")
myFile.close()

... you can also do this:

local myFile = package.open("somefile", "w")
myFile.writeLine("some text")
myFile.close()

Text-mode handles get a slightly better compression ratio than binary, though unless you're planning on writing a decent amount of data to disk there's not all that much of a difference.

#6 Dahknee

  • Members
  • 1,808 posts
  • Location/home/da

Posted 28 April 2015 - 03:04 PM

View PostBomb Bloke, on 29 January 2015 - 06:30 AM, said:

- snip -
I tried using this to pack a load of files and folders together and it just errors with:
package:85: too many results

I used:
package fileput omi test


#7 Bomb Bloke

    Hobbyist Coder

  • Moderators
  • 7,099 posts
  • LocationTasmania (AU)

Posted 29 April 2015 - 12:16 AM

Huh.

What it's saying is that one of your files - not sure which, but it should've printed its name before it errored - is producing too many characters for either unpack() or for string.char() to handle. Probably the former.

Thing is, I can't reproduce that, even when handing in a table with ten million elements (and I'm willing to bet all your files combined wouldn't generate a table even a tenth of that size, bearing in mind that that area of code is only handling individual files...).

What version of ComputerCraft are we talking about, and would you consider uploading the problematic file somewhere?

#8 Dahknee

  • Members
  • 1,808 posts
  • Location/home/da

Posted 29 April 2015 - 08:37 AM

View PostBomb Bloke, on 29 April 2015 - 12:16 AM, said:

Huh.

What it's saying is that one of your files - not sure which, but it should've printed its name before it errored - is producing too many characters for either unpack() or for string.char() to handle. Probably the former.

Thing is, I can't reproduce that, even when handing in a table with ten million elements (and I'm willing to bet all your files combined wouldn't generate a table even a tenth of that size, bearing in mind that that area of code is only handling individual files...).

What version of ComputerCraft are we talking about, and would you consider uploading the problematic file somewhere?

I am using 1.6 currently... on an emulator (everything else works) and I can do?

#9 Bomb Bloke

    Hobbyist Coder

  • Moderators
  • 7,099 posts
  • LocationTasmania (AU)

Posted 28 July 2017 - 01:31 PM

Been a while since I've actually advertised an update for this, but there's some new functionality in 1.6.0 that might be of use about the place.

Personally I've gotten used to developing code in SSP, where I've configured ComputerCraft to up drive sizes considerably. Playing online recently I hence felt severely cramped by the default allowance of a singular manufacturer's megabyte, and turned to BBPack to alleviate the strain.

Loading BBPack (either by simply running it or by loading it as an API) now implements an fs wrapper. By default, the only visible change to your system is the addition of a RAM drive (dubbed "ram", accessible from the root of your HDD) - you can stick pretty much whatever you like in there, although as nothing within it is actually written to disk, all its content will be lost should your system shut down.

BBPack's wrapper can implement additional features via a few commands:

bbpack compress / bbpack.fileSys(true)
These commands iterate through all content on your computer's HDD and compresses it. Files compressed in this manner take on an additional .bbp extension, which is hidden from you so long as BBPack is loaded (the only visible effect is hence a reduction in filesize). Additional files written to your drive in future will also be automatically compressed, although files on external drives are unaffected. In practise I find this increases the capacity of a system by about 3x on average.

startup, .settings, bbpack and its config file (.bbpack.cfg) are unaffected by compression, as are any GIF files.

bbpack decompress / bbpack.fileSys(false)
Reverts filesystem compression, decompressing all files. May leave some files compressed if it runs out of space during the decompression process, in which case the command line version will print a report, or the function version will return false.

bbpack cluster <clusterName>
The system will become a NAS, opening attached modems for rednet usage and granting other systems access to its storage. Multiple systems configured to join the same cluster will share their space, allowing for simple storage of much more data than a single system could hold.

bbpack mount <clusterName> / bbpack.fileSys(<clusterName>)
Opens attached modems for rednet usage and mounts the specified cluster under a local directory of the same name. A cluster can be demounted by rm / fs.delete()ing it. Cluster-stored content is compressed regardless as to the setting for the local drive. Clusters may be mounted by multiple systems at a time. If a server system is added to / removed from a cluster, client systems must be instructed to remount that cluster.

bbpack mount <URL> <path> / bbpack.fileSys(<URL>, <path>)
Mounts the specified web path as a local file, without actually storing it on your local drive. Only handles single files at a time. The mount can be removed by rm / fs.delete()ing it, but is otherwise treated as read-only.

BBPack uses .bbpack.cfg to remember mounts, and to track whether local filesystem compression is active. It'll automatically recall this information when reloaded (eg after a reboot). It's recommended to add os.loadAPI("bbpack") to the very top of your startup file should you wish a given system to use the compression or mounting features. Cluster servers need only shell.run("bbpack cluster <clusterName>").

If BBPack is configured to mount any cluster servers, it'll automatically open any attached modems whenever it's loaded.

Reading from / writing to locally compressed files, or files in a cluster, will cause your system to yield. In particular, shell autocomplete becomes quite slow while browsing clusters - consider disabling autocomplete if this is a problem for you.

Compressed files are difficult to access via external editors. If you're looking for more storage in a single-player environment, you may be better off increasing filesystem sizes within ComputerCraft.cfg.

Edited by Bomb Bloke, 28 July 2017 - 01:33 PM.


#10 SquidDev

    Frickin' laser beams | Resident Necromancer

  • Members
  • 1,427 posts
  • LocationDoes anyone put something serious here?

Posted 28 July 2017 - 04:43 PM

Ooooh, wow. I've seen people talk on the forums before about disk compression, but never seen a working implementation. This is very nice! I'm now intrigued what's taking up all the space - I've only ever hit the limit on my development "computer", never in-world.

Is there ever a time where compressing files would be increase file size and, if so, how does it handle it? I'm just thinking about files which may already be compressed in some form or another.

View PostBomb Bloke, on 28 July 2017 - 01:31 PM, said:

In particular, shell autocomplete becomes quite slow while browsing clusters - consider disabling autocomplete if this is a problem for you.
I wonder if you could put auto-completion in a child coroutine, so it doesn't block input but still provides (albeit delayed) suggestions? Obviously you'd have to override read as well, which isn't ideal.

if file.writeLine or term.setPaletteColour then
	file.write(content)
else
Well, that's an interesting way of determining version. It's nice to see people using the extended binary functions though.

Edited by SquidDev, 28 July 2017 - 04:44 PM.


#11 Bomb Bloke

    Hobbyist Coder

  • Moderators
  • 7,099 posts
  • LocationTasmania (AU)

Posted 29 July 2017 - 02:25 AM

View PostSquidDev, on 28 July 2017 - 04:43 PM, said:

I'm now intrigued what's taking up all the space - I've only ever hit the limit on my development "computer", never in-world.

Basically the fact that I can wget directly from imgur now. Although I did mention that the compression system intentionally ignores GIFs (because they already use LZW compression - hence why my GIF-handling API requires this API!), I've rigged up BLittle to process whole animations, and it most certainly helps with the BLT files that outputs.

Plus Cranium's NBS collection adds up to over a meg just on its own. I've rigged up a single download pack for that, by the way.

View PostSquidDev, on 28 July 2017 - 04:43 PM, said:

Is there ever a time where compressing files would be increase file size and, if so, how does it handle it?

Files which're already sufficiently dense may indeed get larger when compressed, in which case BBPack will simply go ahead and save the compressed versions anyway - it doesn't perform any comparisons (though since you mention it, I can see a spot where it should). The single current allowance I've made for this is the filetype blacklist system, to which I can easily add more entries. Hmm. I suppose I should've added ZIPs.

However, most types a user would actually save to a ComputerCraft system (ie text) should see a net reduction. It is worth noting again that startup isn't compressed - a lengthy startup file is hence better off split into a "just load BBPack and then shell.run() whatever else you want" version.

View PostSquidDev, on 28 July 2017 - 04:43 PM, said:

I wonder if you could put auto-completion in a child coroutine, so it doesn't block input but still provides (albeit delayed) suggestions? Obviously you'd have to override read as well, which isn't ideal.

Overriding read is the last thing I want to do, especially with the constant stream of people trying to change the mainline version's behaviour.

... or, maybe I should try and push a coroutine-based version into mainline myself. Mwuaha.

More likely I'll leave it for people to figure out on their own. Using a GUI-based file manager would skirt the issue nicely, for example.

View PostSquidDev, on 28 July 2017 - 04:43 PM, said:

Well, that's an interesting way of determining version.

I'd prefer to be able to use something like the _HOST constant, but it's not terribly backwards compatible and I don't even trust that it'll be forwards compatible. _CC_VERSION certainly wasn't.

View PostSquidDev, on 28 July 2017 - 04:43 PM, said:

It's nice to see people using the extended binary functions though.

And it's very nice to have them! I'll likely never know why they were omitted in the first place.

#12 Bomb Bloke

    Hobbyist Coder

  • Moderators
  • 7,099 posts
  • LocationTasmania (AU)

Posted 07 September 2017 - 12:59 PM

SquidDev pointed out how easy it is to get the directory listing of a GitHub repo, so BBPack can mount those as virtual drives now (in addition to the single web files and other things it could mount before). Eg:

bbpack mount https://github.com/alekso56/ComputercraftLua/tree/master/treasure gamez
cd gamez/dan200/alongtimeago/
alongtimeago

You can mount different branches / commits / sub-folders / whatever - just head to the point you want using your browser, and copy/paste the URL. In code, you can also produce mounts using "bbpack.fileSys(url, driveName)".

All web-mounts are read-only, but you can unmount them by rm / fs.delete()'ing them. If you copy their contents you'll end up with a local version you can edit (eg "cp gamez localGamez"). Directory listings for repos are only refreshed when you first mount them (or the next time BBPack is loaded after a reboot - I recommend sticking os.loadAPI("bbpack") at the top of your startup file), but file contents are pulled fresh from the servers every time you access them.





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users