Jump to content




Grin - Github Release INstaller


37 replies to this topic

#1 ElvishJerricco

  • Members
  • 803 posts

Posted 30 September 2014 - 06:35 AM

GRIN

Github Release INstaller

Grin is a tool designed to make github based distribution simple. With grin, you can package a release together in a zip file and distribute that to CC users.

Github
Pastebin VuBNx3va

How it works
Unfortunately, there's a bug in LuaJ that prevents CC from getting binary data properly, so normal zips won't quite work. But we do want to use zip files! So to get around this, releases have to be zipped, then encoded with base64. On unix/linux/mac, use the base64 command line utility. On windows, use the certutil command line utility. Once you have your desired release packaged up, create a release on github, and attach the .zip.base64 file to the release.

That's it
That's all you need to do. Now any user can use grin -u YourUsername -r YourRepo dir to download, decode, and unarchive your release to the desired directory.

Tags
Grin takes an optional argument "-t tag_name" that specifies the tag of a release to download, if you don't want to download just the latest release.

Custom Installers
Of course, you can (and probably should) create your own installer that calls grin from pastebin with specified arguments. For example

shell.run("pastebin", "run", "VuBNx3va", "-u", "YourUsername", "-r", "YourRepo", "desiredDir")
print("Installed")

That's all it takes. Put your equivalent of that code on a pastebin and all your users have to do to install your project is pastebin run ID.

But there's now a new flag that lets your custom installers look how you like. The -emit-events (or -e) flag can be used to tell grin to emit events with progress messages instead of printing them. This allows you to do a custom view over Grin.
local ok, err
parallel.waitForAny(function()
	ok, err = pcall(shell.run, "grin", "-e", "-u", user, "-r", repo, dir)
end, function()
	while true do
		local e, s = os.pullEvent("grin_install_status")
		local x, y = term.getCursorPos()
		term.setCursorPos(1, y)
		term.clearLine()
		term.write(s)
	end
end)

assert(ok, err)
local x, y = term.getCursorPos()
term.setCursorPos(1, y)
term.clearLine()
print(user, "/", repo, " successfully downloaded")

A Note About Transparency

Grin's main feature is transparency. It's easy to install Grin packages, and users don't even have to know they're using Grin. It's really easy to make an installer that invokes Grin to install your package, and that installer can just be put on pastebin. Letting users do pastebin run CODE to get your program installed is a big plus. And when you have a big project, that's not always trivial to do. And all this comes with the added benefits of handling versioning seamlessly, and enabling binary downloads. I hope Grin will help us solve installations problem.

Edited by ElvishJerricco, 03 December 2014 - 06:16 AM.


#2 ElvishJerricco

  • Members
  • 803 posts

Posted 01 October 2014 - 07:33 AM

Is there interest in this growing into more of a package management tool? Installs could include a .grin directory with files that represent dependancies to download and install-time configurations to be made.

#3 Lignum

  • Members
  • 558 posts

Posted 03 October 2014 - 02:30 PM

View PostElvishJerricco, on 01 October 2014 - 07:33 AM, said:

Is there interest in this growing into more of a package management tool? Installs could include a .grin directory with files that represent dependancies to download and install-time configurations to be made.
You would create a DLL Hell in ComputerCraft by doing that but I think it's for the better. So go ahead! I'd love to see that feature.

Edited by Lignum, 03 October 2014 - 02:31 PM.


#4 ElvishJerricco

  • Members
  • 803 posts

Posted 03 October 2014 - 04:04 PM

View PostLignum, on 03 October 2014 - 02:30 PM, said:

You would create a DLL Hell in ComputerCraft by doing that but I think it's for the better. So go ahead! I'd love to see that feature.

Well I ended up doing it (a bit differently) in a separate program based on this, Grin-Get. Uses Grin to perform the downloads while Grin-Get manages the packages and dependencies and whatnot. So Grin packages can be installed with Grin or Grin-Get, and packages can include Grin-Get specific features such as dependencies or use of the grin API.

#5 Lignum

  • Members
  • 558 posts

Posted 03 October 2014 - 06:45 PM

View PostElvishJerricco, on 03 October 2014 - 04:04 PM, said:

Well I ended up doing it (a bit differently) in a separate program based on this, Grin-Get. Uses Grin to perform the downloads while Grin-Get manages the packages and dependencies and whatnot. So Grin packages can be installed with Grin or Grin-Get, and packages can include Grin-Get specific features such as dependencies or use of the grin API.
Ah, yes, I've seen grin-get already earlier today. However I didn't notice that you had already implemented it. I'll be sure to try it out once I need it.

#6 ElvishJerricco

  • Members
  • 803 posts

Posted 06 October 2014 - 03:32 AM

Just added a new feature. Installers can now create custom displays.

#7 ElvishJerricco

  • Members
  • 803 posts

Posted 10 October 2014 - 11:23 AM

Update. Changes:
  • Calls error() instead of just using printError and return.


#8 ElvishJerricco

  • Members
  • 803 posts

Posted 03 December 2014 - 06:18 AM

Updated: Transferred ownership to the new team Yevano and I have started.

#9 FUNCTION MAN!

  • Members
  • 292 posts

Posted 03 December 2014 - 11:00 AM

I have noticed that you are using the releases to download repositories. I would suggest being able to use the Github API to download repositories instead of limiting them to releases.
The Github API provides JSON-based responses to http GET queries. Say, to get the latest commit of repository ccLinux/kernel (click click) you'd GET https://api.github.c...fs/heads/master and then get the value at response['object']['sha']. To get all files in that same repository in that commit, GET https://api.github.c...rnel/git/trees/ee86ca54c547430915c93f6853b7b296432fc78b?recursive=1. You can test these in your browser, too.


TL;DR: Some people don't like releasing base64 encoded zipfiles, so, including a Github API-based option to download the whole repository in the latest commit would be nice. kthxbye.

#10 ElvishJerricco

  • Members
  • 803 posts

Posted 03 December 2014 - 06:21 PM

View PostXepoof, on 03 December 2014 - 11:00 AM, said:

I have noticed that you are using the releases to download repositories. I would suggest being able to use the Github API to download repositories instead of limiting them to releases.
The Github API provides JSON-based responses to http GET queries. Say, to get the latest commit of repository ccLinux/kernel (click click) you'd GET https://api.github.c...fs/heads/master and then get the value at response['object']['sha']. To get all files in that same repository in that commit, GET https://api.github.c...rnel/git/trees/ee86ca54c547430915c93f6853b7b296432fc78b?recursive=1. You can test these in your browser, too.


TL;DR: Some people don't like releasing base64 encoded zipfiles, so, including a Github API-based option to download the whole repository in the latest commit would be nice. kthxbye.

I know that you can download repos. That completely defeats the purpose of Grin. Grin has two main advantages. One is that the download system is efficient, and the other is that it differentiates the dev and user environments.

JVML for example, has to distribute a jar file. CC has a bug from LuaJ that makes downloading binaries impossible. So encoding to base64 is the only way to fix that. While we're at, we might as well support packaging the release nicely. That's why Grin uses .zip.base64. This way you can archive exactly what you need into the release and it works with everything.

As for how it differentiates the dev and user environments, if you have a large project, then there's a lot of files that you need in your source repo that you don't want users to have to have on their system. For example, if you need to do any kind of compiling, you don't want to be releasing the source and telling the user they have to do the compiling. You just want to release the compiled thing and only the compiled thing. You also likely don't want your test scripts to be in your release. So basically, a developer wants the ability to dictate exactly what makes it to an end users system, and archiving releases like this is the way to do that. Just as an example, JVML's released version is ⅓ the size of its source code version. That's huge.

Finally, I don't know about anyone else, but I don't like releasing unstable software to users. If they're pulling directly from the source code, they can catch a commit mid-development. That's not good. It's important to me that I can dictate exactly when and how a release is.

All the above things are the problems are the things Grin is here to solve. If you want to download a github repo, use the clone button. That's not a proper way to do releases.

Edited by ElvishJerricco, 03 December 2014 - 06:23 PM.


#11 MarioBG

  • Members
  • 43 posts
  • LocationSpain (or Hisperia)

Posted 15 December 2014 - 05:41 PM

Hello, I'm rather new to GitHub (in fact, I've started to use it due to this program). It would help me a lot if you could patiently explain to me how do I have to set the thing up in GitHub so the files I have in my computer can be translated to another computer without any kind of change. Up to this moment, all I have managed is a string:1626:malformed response error. Thanks a lot.
If it helps, the repository is at https://github.com/MarioBG/Olive_OS. Thanks again, and I hope we can fix this.

EDIT: I made some progress, and now, if I try with another repository I made (https://github.com/M...G/Mar-Mar-Olive), it shows this Posted Image. Please help. Also, if you have a peek at the actual OS, I want to say this is NOT the final 1.2 version, just some kind of prerelease, but i didn't want to label it as such in fear it would not work.

Edited by MarioBG, 15 December 2014 - 06:16 PM.


#12 ElvishJerricco

  • Members
  • 803 posts

Posted 15 December 2014 - 08:27 PM

Something is wrong with the way you're encoding the base64 file or the zip file. How exactly are you doing it?

#13 MarioBG

  • Members
  • 43 posts
  • LocationSpain (or Hisperia)

Posted 16 December 2014 - 06:52 AM

View PostElvishJerricco, on 15 December 2014 - 08:27 PM, said:

Something is wrong with the way you're encoding the base64 file or the zip file. How exactly are you doing it?
Posted Image
In the Olive zip, I had previously stored the content of my root folder from the computer in which I store the OS.

Edited by MarioBG, 16 December 2014 - 06:53 AM.


#14 ElvishJerricco

  • Members
  • 803 posts

Posted 16 December 2014 - 07:08 AM

Oh wow. I'd never used certutil before. I'm a UNIX guy, and haven't even tested certutil. I opened your base64 file in a text editor and it has this header and footer and is separated into lines. That's not correct. It should just be a string of ascii characters, with no new lines and no header or footer. It would appear certutil is not the right way to encode base64 on windows.

#15 SquidDev

    Frickin' laser beams | Resident Necromancer

  • Members
  • 1,427 posts
  • LocationDoes anyone put something serious here?

Posted 16 December 2014 - 05:45 PM

Bits of me thing there should be a pure Lua way of zipping the file. I know the deflate library doesn't support it and I'm sure it would be very hard to write your own. I might do it if I have the time.

#16 MarioBG

  • Members
  • 43 posts
  • LocationSpain (or Hisperia)

Posted 16 December 2014 - 06:20 PM

Alright, alright, we've made some progress. It now gets to download the file and recognise its content. It scans the zip and starts unarchiving, but at the very first file, each time I try... (note: I'm working on this repository)Posted Image
In the middle one I tried again with certutil, but besides generating those certificate thingies, it also splits the file in lines, which makes it kind of impossible to understand for Grin, I suppose.

EDIT 2: Alright, THAT particular problem was because the computer had been set to have 0B HDD XD. Now I've somewhat progressed, and it does unpack some items, but when it gets to the .lua file extension data, it gives this error.
Posted Image
The contents of the file are as follow:
action_check
2048
L
I honestly don't get why that would pose a problem, but apparently it does. Thanks for your patience. Also, would there be any chance, slight as it may be, to rework one or two bits of this to make it compatible with CC 1.48? It would make a great difference to me, since the main feature I made this OS for was to have something for my 1.4.7 server (and then it got a bit out of control, I guess). Thanks a huge lot!

Edited by MarioBG, 16 December 2014 - 08:56 PM.


#17 ElvishJerricco

  • Members
  • 803 posts

Posted 17 December 2014 - 06:49 AM

View PostMarioBG, on 16 December 2014 - 06:20 PM, said:

-snip-

Ok I've found the problem. I feel silly now =P I wrote the wrong variable name in the code that handles files that have no compression, and apparently your .lua had no compression. Fixed

EDIT: Also, what's not working in 1.48?

EDIT 2: Also I would like to know how you're encoding base64 now. I should updated my scripts to not use certutil.

Edited by ElvishJerricco, 17 December 2014 - 06:53 AM.


#18 ByteMe

  • Members
  • 124 posts
  • LocationAdelaide, Australia

Posted 17 December 2014 - 08:02 AM

The DLL would be very slow.

#19 MarioBG

  • Members
  • 43 posts
  • LocationSpain (or Hisperia)

Posted 17 December 2014 - 03:01 PM

View PostElvishJerricco, on 17 December 2014 - 06:49 AM, said:

View PostMarioBG, on 16 December 2014 - 06:20 PM, said:

-snip-

Ok I've found the problem. I feel silly now =P I wrote the wrong variable name in the code that handles files that have no compression, and apparently your .lua had no compression. Fixed

EDIT: Also, what's not working in 1.48?

EDIT 2: Also I would like to know how you're encoding base64 now. I should updated my scripts to not use certutil.

After the "Decoding Base64" message, it outputs grin:1376: too few arguments. I've had my issues of the sort in the OS itself. If it could be fixed though, it would be awesome :).

I use an online tool now and paste the text onto the file. Not perfect, but it works. The tool is this one.

Edited by MarioBG, 17 December 2014 - 03:04 PM.


#20 ElvishJerricco

  • Members
  • 803 posts

Posted 17 December 2014 - 04:17 PM

View PostMarioBG, on 17 December 2014 - 03:01 PM, said:

-snip-

That tool you're using prefixes the output with
data:;base64,
Make sure you're only copying what comes after that comma. I believe that is what's causing your error.

View PostByteMe, on 17 December 2014 - 08:02 AM, said:

The DLL would be very slow.

And what are you talking about?

Edited by ElvishJerricco, 17 December 2014 - 04:15 PM.






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users