Jump to content




NodeQuarry - A Cheap, Scalable, Web-Based Turtle Quarry

turtle networking lua

47 replies to this topic

#1 apemanzilla

  • Members
  • 1,421 posts

Posted 14 November 2015 - 04:49 AM

NodeQuarry is a special turtle quarry program. It is designed to efficiently extract ore, while keeping costs low. It uses a node.js program as a host, meaning you don't need to rely on rednet.

Please note that NodeQuarry is in beta! There will be bugs!

Posted Image

Features
  • No 'host' turtle required
  • No rednet modems required (a miner only requires a pickaxe and nothing more)
  • You can use as many turtles as you want in a single swarm, all working together to extract ores
  • If given a bucket, turtles can refuel themselves from lava lakes to reduce fuel costs
  • "Junk" blacklist - turtles will not mine worthless blocks like cobblestone and dirt, saving time and inventory space.
  • Persistence - if the chunk unloads, or the world is reloaded, the turtles will resume right where they left off.
Instead of using modems like conventional turtle quarry programs, it uses a node.js server, to bypass issues such as limited rednet range, and channel clutter. You can mine quarries of any size you want with as many or as few turtles as you want - all you need to use it is:
  • A real computer to run the host software - for singleplayer worlds, you can use the same PC you play Minecraft on. For multiplayer, you'll need to do some portforwarding, or use a hosted server. You don't need much processing power at all, it's a very lightweight program.
  • Mining turtles. You don't need modems on them, just pickaxes. You can have as few as one or as many as one hundred or more.
  • Chests, or other forms of storage. You will need at least two - one to store fuel for the turtles, and one for the turtles to drop ore in.
  • A disk drive and floppy disk. These are used to quickly deploy turtles into the swarm.
Great, now that you have all that, let's get to setting it up!

Host Server

Turtle Swarm


Once the turtles finish mining an area, they will, one by one, return to the dropoff area and drop all their loot into the chests. They will then wait there, and the next turtle will break and put them in the chest, until there is only one turtle left. The turtles will also uninstall the swarm programs from themselves so they are ready to be used for something else, or maybe another swarm.

The source code for the turtle code is available here, and the source code for the host server is available here.

Try out the program and let me know of any bugs you find or features you want!

Edited by apemanzilla, 14 November 2015 - 05:00 AM.


#2 underspikey

  • Members
  • 4 posts

Posted 14 November 2015 - 07:36 PM

Hey! Quick note, the file in the .zip is actually swarm-quarry-host-master.
Might want a bat file for that anyway :P

#3 apemanzilla

  • Members
  • 1,421 posts

Posted 14 November 2015 - 09:14 PM

View Postunderspikey, on 14 November 2015 - 07:36 PM, said:

Hey! Quick note, the file in the .zip is actually swarm-quarry-host-master.
Might want a bat file for that anyway :P

That's intended. The zip contains the host code, if you want to set up the turtle code then you run the installer from pastebin. You need the HTTP API for this to work at all anyways, so it's assumed that you can use the pastebin program.

#4 underspikey

  • Members
  • 4 posts

Posted 14 November 2015 - 11:11 PM

View Postapemanzilla, on 14 November 2015 - 09:14 PM, said:

View Postunderspikey, on 14 November 2015 - 07:36 PM, said:

Hey! Quick note, the file in the .zip is actually swarm-quarry-host-master. Might want a bat file for that anyway :P/>
That's intended. The zip contains the host code, if you want to set up the turtle code then you run the installer from pastebin. You need the HTTP API for this to work at all anyways, so it's assumed that you can use the pastebin program.

Yeah, I meant the host program.
In your tutorial you tell to use the "cd swarm-quarry-host", but if you don't clone the rep and zip it instead, it adds the branch name, so you have to "cd swarm-quarry-host-master".

Running the program with 2 turtles atm (infinity evolved turtles are expensive as all hell, had to cheat in a computer to get it started at all :P)
I had a little problem getting it to work. I followed the instructions step-by-step, looking at the pictures you posted (quite new to ComputerCraft), but placing the turtle on top of the disk didn't prompt me. I tried running the setup again, with "cd disk" before and it ran then. Dunno if that was intended, just pointing out.
Also, is there a way to extend the blacklist? Getting a lot of limestone and marble and other useless crap like that.

OK, got the first iteration done now. I made it do a 16x16 to begin with, just to test the waters. A few things here:
It didn't really start the counting from the block it was on, instead it moved two spaces forward. Not a huge problem, just have to make the length shorter. Also for the chests. Oh, and because it checks for blocks wider than the actual perimeter.
Then I went on to see how big the area actually was. It did a 17x17. It checked a 19x19. Just things to consider again, not a problem really.
When the second miner ended it's cycle, it mined the previous miner up, which is fine. However, I had a bucket in that miner and it didn't pick that up. Wasn't in the chest either, so I guess you forgot to let it check and put away the bucket before uninstall?
Also, checked the code a bit, noticed you had actually blacklisted the chisel blocks, but I think it's not working, my chests are getting clogged by them.
Those huge holes are also getting a bit crazy, is there a way you can make the turtle close them after a shaft is complete?

Now, I'd like to increase the size to a 3x3 chunk(max any world anchor can handle in this modpack). According to my calculations, that would mean a 45*39 in blocks, if I also count all the extra space it needs. So as I understand I need to create a new swarm. So I started doing that, created a new floppy disk and tried to pastebin the code again, and I get up to the point until it has prompted me all the questions (host, name, width, length) and is about to finish when it throws an "File exists" error. Too tired to try to figure out what that means atm.

#5 apemanzilla

  • Members
  • 1,421 posts

Posted 15 November 2015 - 03:05 AM

View Postunderspikey, on 14 November 2015 - 11:11 PM, said:

View Postapemanzilla, on 14 November 2015 - 09:14 PM, said:

View Postunderspikey, on 14 November 2015 - 07:36 PM, said:

Hey! Quick note, the file in the .zip is actually swarm-quarry-host-master. Might want a bat file for that anyway :P/>
That's intended. The zip contains the host code, if you want to set up the turtle code then you run the installer from pastebin. You need the HTTP API for this to work at all anyways, so it's assumed that you can use the pastebin program.

Yeah, I meant the host program.
In your tutorial you tell to use the "cd swarm-quarry-host", but if you don't clone the rep and zip it instead, it adds the branch name, so you have to "cd swarm-quarry-host-master".

Running the program with 2 turtles atm (infinity evolved turtles are expensive as all hell, had to cheat in a computer to get it started at all :P)
I had a little problem getting it to work. I followed the instructions step-by-step, looking at the pictures you posted (quite new to ComputerCraft), but placing the turtle on top of the disk didn't prompt me. I tried running the setup again, with "cd disk" before and it ran then. Dunno if that was intended, just pointing out.
Also, is there a way to extend the blacklist? Getting a lot of limestone and marble and other useless crap like that.

OK, got the first iteration done now. I made it do a 16x16 to begin with, just to test the waters. A few things here:
It didn't really start the counting from the block it was on, instead it moved two spaces forward. Not a huge problem, just have to make the length shorter. Also for the chests. Oh, and because it checks for blocks wider than the actual perimeter.
Then I went on to see how big the area actually was. It did a 17x17. It checked a 19x19. Just things to consider again, not a problem really.
When the second miner ended it's cycle, it mined the previous miner up, which is fine. However, I had a bucket in that miner and it didn't pick that up. Wasn't in the chest either, so I guess you forgot to let it check and put away the bucket before uninstall?
Also, checked the code a bit, noticed you had actually blacklisted the chisel blocks, but I think it's not working, my chests are getting clogged by them.
Those huge holes are also getting a bit crazy, is there a way you can make the turtle close them after a shaft is complete?

Now, I'd like to increase the size to a 3x3 chunk(max any world anchor can handle in this modpack). According to my calculations, that would mean a 45*39 in blocks, if I also count all the extra space it needs. So as I understand I need to create a new swarm. So I started doing that, created a new floppy disk and tried to pastebin the code again, and I get up to the point until it has prompted me all the questions (host, name, width, length) and is about to finish when it throws an "File exists" error. Too tired to try to figure out what that means atm.

1) Ah, I see. I didn't bother creating a batch file because installation is only one command, and running it is also one command.

2) I believe that is a CC bug - I have noticed that in creative mode, if I shift-right click a disk into a drive it wipes (?) the disk. If I manually open the drive and place the disk in, it works. The "cd disk" part will not affect anything. If this happens again, can you quickly do "ls disk" to see if the disk got wiped or something?

3) Right now the blacklist is hardcoded. I intend to add a step in the setup where you can (on a turtle) give it blocks it should ignore. I recently added a few mod blocks to the list, but if more are missing, submit an issue or pull request here. The blacklist is on line 7 of the miner api.

4) The entire quarry starts two blocks forward from the disk drive. This is to prevent the turtles mining the disk drive itself. It should still be 16x16 total.

5) I don't quite understand what you mean by the 17x17 becoming 19x19 part - could you upload some screenshots?

6) The bucket being left behind is a bug. The turtles should drop their buckets into the dropoff chest when mining is complete so that none are left behind.

7) The new blocks added to the blacklist will only take effect if you created the swarm installation disk after I posted the new code. If it's still not working as intended I'll check and fix it.

8) A chunk is 16x16 - 3 chunks should be at minimum, 48x48. With the extra blocks for the storage and disk drive, that should be 48 blocks wide and 53 blocks deep. (You still put 48 and 48 into the setup program - it adds the extra blocks internally)

9) This is actually a bug when you re-run the setup with a disk that already has the swarm installer - the setup does not delete the old files on the disk. I will be fixing this, but it shouldn't have happened if it was a new disk anyways.

Thanks for all the feedback! I'll fix the mentioned bugs and post when done.

Edit: I have patched the bugs with the bucket and with the "File exists" error in the setup. You will need to create a new swarm for these changes to take effect. If you still encounter them, please let me know.

Edited by apemanzilla, 15 November 2015 - 03:11 AM.


#6 underspikey

  • Members
  • 4 posts

Posted 15 November 2015 - 12:07 PM

Ok, so I asked for a 16x16 and here's what it actually made:
Posted Image
The piece of iron ore is where the drive used to be. As you can see, the holes themselves form a 17x17. But for chunkloading purposes, If we for example come and check a hole in the corner, we see that it checks all four directions.
Posted Image
Which means that a 17x17 workfield becomes a 19x19 field to be chunkloaded.

Ok, I understand the shift-rightclick wipe bug now, that does make things easier!
Now, got the second swarm to work.
I encountered another bug, however. I placed down my turtle on the drive, gave it a bucket and told it to add itself to the swarm. Forgot fuel tho, so i ran back to my base. I think the chunk got unloaded and when I came back, it had reset and asked me whether I wanted to add the miner to the swarm again. If I said yes, then it said file exists etc. I deleted the startup and swarm and tried again, got it to work.

However, I put the second miner on relatively quickly after the first. This resulted in a problem after the first shafts were mined. The first miner went back to unload and started doing so. However, the second miner also came along and broke the first one, dropping the contents of the first miner on the ground(a bucket and ores) and put the miner itself in the chest. I'm glad I only have two turtles now, that would be a lot of buckets lost (buckets are pretty hard to get in infinity evolved)

On a sidenote, about that 48x48. My problem is that I have an achor down and I am not around myself. An anchor allows a 3x3 chunk to be loaded, so everything must happen inside those 3x3 chunks. So I have to calculate down from 48x48. I know that it adds the necessary blocks itself, tho I'm not sure if that's a good thing. You might want to just ask for how much space there is to use, but that's a minor detail.

Also noticed that I did install the miner program just today, a few minutes ago, so it should be up-to-date, but limestone and marble is still being picked up :(

#7 apemanzilla

  • Members
  • 1,421 posts

Posted 15 November 2015 - 06:33 PM

1) The 16x16 to 17x17 is a bug in the server software, I'll fix it. You should not need to chunkload any extra for it checking the sides - the turtle itself does not move out of the shaft, it only turns and digs and that's it. So, if the turtle is in chunk A, and inspects and digs in chunk B, you shouldn't need to chunkload chunk B.

2) Turtle having error copying - I'll fix this, thanks for reporting it.

3) This is a bug that I've been trying to think of a good fix for. Right now, turtles break everything in their way, regardless of what it is. I've been trying to think of a way to make it so that when a turtle detects another turtle (via the peripheral API) it waits/moves so they can pass eachother.

4) Ah, I see what you mean about the 3x3 area. I'll look into adding a setup option where it will lay out the maximum size quarry for given dimensions, including the dropoff area.

5) Alright, I'll look into this too. If possible, can you paste the output of running turtle.inspect with the blocks you don't want in front of the turtle?

#8 underspikey

  • Members
  • 4 posts

Posted 15 November 2015 - 06:53 PM

I noticed that after the last update it really burns through the coal. Not sure why. It did four shafts and six stacks of coal gone. Not sure why... Don\t have enough coal to get the turtle running anymore :P

#9 Brekkjern

  • Members
  • 13 posts

Posted 17 November 2015 - 12:26 AM

I added a pull request that should fix the blacklist bug I think. I'm a bit new with Git, so I might have done it wrong. Give me a shout if you need me to do something.

View Postapemanzilla, on 15 November 2015 - 06:33 PM, said:

3) This is a bug that I've been trying to think of a good fix for. Right now, turtles break everything in their way, regardless of what it is. I've been trying to think of a way to make it so that when a turtle detects another turtle (via the peripheral API) it waits/moves so they can pass eachother.

You could have the turtles move in different height levels depending on if the turtle is entering or exiting the loading station. Alternatively, you could implement a queuing spot where the turtle moves to the queue location and stops if it meets another turtle. It could then query the server and ask for its position in the queue. Just a few ideas on that.

#10 apemanzilla

  • Members
  • 1,421 posts

Posted 17 November 2015 - 12:35 AM

View PostBrekkjern, on 17 November 2015 - 12:26 AM, said:

I added a pull request that should fix the blacklist bug I think. I'm a bit new with Git, so I might have done it wrong. Give me a shout if you need me to do something.

View Postapemanzilla, on 15 November 2015 - 06:33 PM, said:

3) This is a bug that I've been trying to think of a good fix for. Right now, turtles break everything in their way, regardless of what it is. I've been trying to think of a way to make it so that when a turtle detects another turtle (via the peripheral API) it waits/moves so they can pass eachother.

You could have the turtles move in different height levels depending on if the turtle is entering or exiting the loading station. Alternatively, you could implement a queuing spot where the turtle moves to the queue location and stops if it meets another turtle. It could then query the server and ask for its position in the queue. Just a few ideas on that.

Thanks for the PR, I haven't had much time to squash bugs. You missed some commas though, if you fix that I'll accept the PR.

That is a good idea about the pathfinding - I had thought of having them detect turtles and turn right, go forward, turn left and resume pathfinding, but it would be less reliable. I'll look into implementing your suggestion about it.

#11 Brekkjern

  • Members
  • 13 posts

Posted 17 November 2015 - 01:38 AM

View Postapemanzilla, on 17 November 2015 - 12:35 AM, said:

Thanks for the PR, I haven't had much time to squash bugs. You missed some commas though, if you fix that I'll accept the PR.
Fixed them now.

View Postapemanzilla, on 17 November 2015 - 12:35 AM, said:

That is a good idea about the pathfinding - I had thought of having them detect turtles and turn right, go forward, turn left and resume pathfinding, but it would be less reliable. I'll look into implementing your suggestion about it.

Yeah. I tinkered with some pathfinding myself at some point. It's very easy to end up with designs that can go into infinite loops. Utilizing vertical space can help resolve that.

Edited by Brekkjern, 17 November 2015 - 01:39 AM.


#12 apemanzilla

  • Members
  • 1,421 posts

Posted 17 November 2015 - 04:39 AM

View PostBrekkjern, on 17 November 2015 - 01:38 AM, said:

View Postapemanzilla, on 17 November 2015 - 12:35 AM, said:

Thanks for the PR, I haven't had much time to squash bugs. You missed some commas though, if you fix that I'll accept the PR.
Fixed them now.

View Postapemanzilla, on 17 November 2015 - 12:35 AM, said:

That is a good idea about the pathfinding - I had thought of having them detect turtles and turn right, go forward, turn left and resume pathfinding, but it would be less reliable. I'll look into implementing your suggestion about it.

Yeah. I tinkered with some pathfinding myself at some point. It's very easy to end up with designs that can go into infinite loops. Utilizing vertical space can help resolve that.

And accepted. Let me know if there's anything else to add to the blacklist.

I won't have much free time until Wednesday, so I'll do a patch then to improve the aggressive pathfinding and any other bugs.

View Postunderspikey, on 15 November 2015 - 06:53 PM, said:

I noticed that after the last update it really burns through the coal. Not sure why. It did four shafts and six stacks of coal gone. Not sure why... Don\t have enough coal to get the turtle running anymore :P

That's strange - that shouldn't happen unless the shafts are very far from the dropoff point. Can you check that the coal isn't being dropped in the ore chest by mistake? It may be a side effect of the bucket patch.

Edited by apemanzilla, 17 November 2015 - 04:40 AM.


#13 Brekkjern

  • Members
  • 13 posts

Posted 17 November 2015 - 01:49 PM

Just to add something more to your to-do list. Maybe it would make sense to have the blacklist hosted on the server and have the turtle download the list when it is assigned a shaft. That way you can add more things to the blacklist while the swarm is running. I'm not sure how much work it is as I don't know JS and certaintly not Node.JS, but that would expand the feature set a bit.

#14 apemanzilla

  • Members
  • 1,421 posts

Posted 17 November 2015 - 01:57 PM

View PostBrekkjern, on 17 November 2015 - 01:49 PM, said:

Just to add something more to your to-do list. Maybe it would make sense to have the blacklist hosted on the server and have the turtle download the list when it is assigned a shaft. That way you can add more things to the blacklist while the swarm is running. I'm not sure how much work it is as I don't know JS and certaintly not Node.JS, but that would expand the feature set a bit.

I've been considering a system like this, although I prefer to (for consistency) not modify the swarm configuration after it is created.

I think I'll move the blacklist to the configuration rather than hardcode it, then add a step in the setup that, when run on turtles, allows you to give the turtle extra blocks to ignore.

#15 maximuscr31

  • New Members
  • 1 posts

Posted 17 November 2015 - 05:37 PM

I don't wish to come across as incompetent. I downloaded node.js and installed it. I can't seem to get past that step. I have opened up node js and tried the commands listed. I get errors. I tried command prompt and get errors. I try right clicking and using node.js to open the swarm file. It closes out immediately. I can't get git to work neither. Could I get some more dumbed down step by steps? I am not sure what stupidity I am experiencing but I would love to give this a try. I am on windows 10 with nodejs 64.

#16 apemanzilla

  • Members
  • 1,421 posts

Posted 18 November 2015 - 04:46 AM

View Postmaximuscr31, on 17 November 2015 - 05:37 PM, said:

I don't wish to come across as incompetent. I downloaded node.js and installed it. I can't seem to get past that step. I have opened up node js and tried the commands listed. I get errors. I tried command prompt and get errors. I try right clicking and using node.js to open the swarm file. It closes out immediately. I can't get git to work neither. Could I get some more dumbed down step by steps? I am not sure what stupidity I am experiencing but I would love to give this a try. I am on windows 10 with nodejs 64.

Don't worry, it's fine.

Download the zip from here and save it somewhere. Then extract it, and open the folders until you see a file called "swarmhost.js". When you see that, go to File > Open command prompt > Open command prompt. This will open a command window in the current folder. In the command prompt, run these two commands, in order:
npm install
node swarmhost.js
The first command installs everything the program needs. It only needs to be run the first time. The second command actually starts the program. If there are no errors, you can proceed to the disk and turtle setup.

In the setup, when you are asked for the "host address", make sure to enter "http://localhost:8080" without the quotes. This tells the program that the server is running on your computer and not a different web server.

Let me know if you still have problems.

#17 apemanzilla

  • Members
  • 1,421 posts

Posted 18 November 2015 - 07:56 PM

A major update has been released for the turtle miner software.

New and improved fuel efficiency features
  • Turtles will now automatically set their label if one is not set to prevent fuel loss when broken.
  • In the initial setup you can now set an extra fuel value. When refueling from the chest, turtles will now take this much extra fuel beyond what is needed. Higher values will cost more fuel initially, but will improve efficiency by allowing turtles to mine multiple shafts in a row instead of refueling after each one. This value defaults to 250 (meaning turtles will take 250 extra fuel when refueling out of the chest)
  • Turtles will attempt to refuel from inventory items before going to and taking fuel from the chest. If they have enough fuel with them, they will not go to the chest to take more fuel at all, allowing them to proceed directly to the next shaft.
During testing of the fuel efficiency changes, a swarm of 5 turtles with default settings mining a 10x10 area consumed 41 pieces of charcoal from the fuel chest and returned 74 pieces of coal that were mined. The turtles were given buckets to scoop lava with.

Improved blacklist
  • The blacklist has been expanded to include a few additional mod blocks, and snow layers and ice.
  • The blacklist has been moved to the setup program. This is in preparation of a customizable blacklist and/or improved setup program.
Improved pathing
  • As suggested by Brekkjern, turtles now operate at different Y levels depending on whether they are going to a shaft to begin mining or going to the dropoff area. This should help prevent cases where turtles will break eachother, dropping items.
These changes will take effect only when you re-download and run the setup program from the original post, making a new swarm. Please let me know if you encounter any bugs or have any problems with the program.

The next update will likely include one or more of the following:
  • Customizable blacklist
  • Further improved turtle pathing and avoidance systems
  • New/improved setup program

Edited by apemanzilla, 18 November 2015 - 08:05 PM.


#18 apemanzilla

  • Members
  • 1,421 posts

Posted 18 November 2015 - 08:07 PM

View Postunderspikey, on 15 November 2015 - 06:53 PM, said:

I noticed that after the last update it really burns through the coal. Not sure why. It did four shafts and six stacks of coal gone. Not sure why... Don't have enough coal to get the turtle running anymore :P

I just released a new update that should help with this. As I was testing it with a 10x10 quarry, 5 turtles (with buckets) brought back more fuel than they used, using 41 charcoal and bringing back 74 coal.

#19 tech

  • New Members
  • 1 posts

Posted 06 December 2015 - 03:26 PM

I created a simple script that automatically deploys turtles.

Place a normal turtle to the right of where you will place the turtles facing the block to place them at.

Run:
pastebin get fxY8Kggc startup
reboot

And also open the disk on a computer and type:
cd disk
pastebin get 4uu8PTfT startup

Now just put some fuel in slot 16 of the normal turtle, and if you put a mining turtle in any other slot, it'll get placed and get right to work digging blocks!

#20 oxygencraft

  • Members
  • 38 posts

Posted 13 December 2015 - 03:01 AM

It is not working. The placeholder text is not turning into a web control panel.





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users