19 replies to this topic

[Wilma456]

What's the best way, to download a full folder from GitHub without writing all raw URLs in the code?

Edited by Wilma456, 04 September 2017 - 12:49 PM.

[SquidDev]

You can get the full tree of a repository by using the trees API endpoint. This returns a JSON object with a property "tree", which can be iterated over. You can then fetch from https://raw.githubusercontent.com/ to download the actual contents.

Here's an example to download all files in the SquidDev-CC/Howl repository. I'm presuming you have a JSON API loaded already:

-- Fetch the file listing
local request = http.get("https://api.github.com/repos/SquidDev-CC/Howl/git/trees/master?recursive=1")
local contents = json.parse(request.readAll())
request.close()

for _, file in ipairs(contents.tree) do
  -- Fetch the actual contents. You can save these files somewhere, put them in an array, etc...
  local request = http.get("https://raw.githubusercontent.com/SquidDev-CC/Howl/master/" .. file.path)
  local handle = fs.open(file.path, "w")
  handle.write(request.readAll()
  handle.close()
  request.close()
end

[Wilma456]

What json API do you use in the example?

[SquidDev]

Wilma456, on 04 September 2017 - 01:19 PM, said:

What json API do you use in the example?

In that case, a purely hypothetical one. You'll probably want ElvishJerricco's API, which uses json.decode instead.

[Wilma456]

This API has not the function "parse"

[SquidDev]

SquidDev, on 04 September 2017 - 01:29 PM, said:

which uses json.decode instead.

[Wilma456]

I had tried this code with the API that you linked:

os.loadAPI("/json")
local request = http.get("https://api.github.com/repos/SquidDev-CC/Howl/git/trees/master?recursive=1")
local contents = json.decode(request.readAll())
request.close()

for _, file in ipairs(contents.tree) do
  local request = http.get("https://raw.githubusercontent.com/SquidDev-CC/Howl/master/" .. file.path)
  local handle = fs.open(file.path, "w")
  handle.write(request.readAll())
  handle.close()
  request.close()
end

But I got the error 9: attempt to index ? (a nil value)

[SquidDev]

Wilma456, on 05 September 2017 - 05:23 PM, said:

But I got the error 9: attempt to index ? (a nil value)

So the recursive file listing also includes directories, so you'll need to check that file.type == "blob" before downloading each file.

Edited by SquidDev, 05 September 2017 - 05:33 PM.

[Wilma456]

Same error, now with line 10

os.loadAPI("/json")
local request = http.get("https://api.github.com/repos/SquidDev-CC/Howl/git/trees/master?recursive=1")
local contents = json.decode(request.readAll())
request.close()

for _, file in ipairs(contents.tree) do
  if file.type == "blob" then
    local request = http.get("https://raw.githubusercontent.com/SquidDev-CC/Howl/master/" .. file.path)
    local handle = fs.open(file.path, "w")
    handle.write(request.readAll())
    handle.close()
    request.close()
  end
end

[SquidDev]

Wilma456, on 05 September 2017 - 05:46 PM, said:

Same error, now with line 10

I'd presume a HTTP request is failing. Maybe try adding some print statements to determine which file is failing, print the error message from the HTTP response, etc... C'mon, I've seen your programs - I know you're capable of basic debugging skills.

[Lupus590]

SquidDev, on 05 September 2017 - 06:03 PM, said:

I know you're capable of basic debugging skills.

[OffTopic] Do we have a tutorial on debugging on the forums yet?

[apemanzilla]

You can use gitget. /shameless plug

[Wilma456]

apemanzilla, on 05 September 2017 - 09:40 PM, said:

You can use gitget. /shameless plug

It looks like that gitget can only download a full repository and not only a folder. In my case I want to download to rom folder of CC from here or older Versions from here (different Tags on GitHub). Only the rom folder. Nothing else (e.g. bios).

[Lupus590]

You could modify gitget.

[apemanzilla]

Wilma456, on 06 September 2017 - 04:10 PM, said:

apemanzilla, on 05 September 2017 - 09:40 PM, said:

You can use gitget. /shameless plug

It looks like that gitget can only download a full repository and not only a folder. In my case I want to download to rom folder of CC from here or older Versions from here (different Tags on GitHub). Only the rom folder. Nothing else (e.g. bios).

Modify line 108 to add an additional check whether the file starts with a given path.

[Wilma456]

With modifyed gitget, I would be able to download the rom folder from the offical Repo. But what's with older Versions. In the Repo of alekso56 they are sorted with Tags, but gitget only suports branches and not tags.

[SquidDev]

Wilma456, on 07 September 2017 - 01:03 PM, said:

With modifyed gitget, I would be able to download the rom folder from the offical Repo. But what's with older Versions. In the Repo of alekso56 they are sorted with Tags, but gitget only suports branches and not tags.

Both should work fine. You can use branches, tags or specific commit hashes with git, and they'll all "just work".

[Wilma456]

I had try "gitget alekso56 ComputercraftLua branch=1.79" but I get a "gitget:74 attempt to index ? (a nil value) "error

[Bomb Bloke]

The usage shown in the gitget script doesn't seem to match the usage shown in the gitget thread: try "1.79" instead of "branch=1.79". Failing that, rig the code to print the URL it's attempting to connect to so you can report it back here.

[Wilma456]

I had now made a file with all files in /rom to solve the Problem for me. And with the list I don't have any dependencies to gitget or a json API. If you are intrested, you can find the file here