Jump to content




LyqydNet Rednet API

api wireless networking

  • You cannot reply to this topic
104 replies to this topic

#1 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 13 May 2012 - 05:00 AM

So, I've been working on a rednet automatic networking API for a good little while now. It does automatic packet routing, peer discovery, and best-route finding. It can transport packets without loss across unloaded chunks whenever someone runs through the usually-unloaded areas, and it makes it very easy to write robust client-server applications. It is still under development, so feedback and bug reports are most welcome. It has been heavily tested and I use it extensively in my singleplayer world. This has not been tested on multiplayer.

Features:
  • Packet routing/forwarding
  • Transport packets across unloaded chunks
  • Automatic connection handling
  • Terminal redirection across rednet
  • Remote shell program
  • Rednet "repeat" program functionality on routers
  • Route packets efficiently between different networks
This API adds seamless support for background rednet event-activated server applications using coroutines. Complete documentation is included in the readme file. I am planning on writing up an auto-installation program to make things easier for those on multiplayer servers who don't have access to the actual computer files. No hard modification of the ROM files is necessary to run this.

I will be cleaning up and releasing the various applications I've written that utilize this, including a file server, remote shell, remote vault, a compatible version of the GPS program, and an in-progress chat program, over the next few weeks.

The best way to get LyqydNet is via packman, with the command packman install lyqydnet, as this will fetch LyqydNet automatically. You could also grab it from github.

Hope this helps!

#2 tirankenja

  • New Members
  • 4 posts

Posted 20 May 2012 - 10:18 AM

Hey. I've been playing a little with this. But for some reason I am getting a crash when I try to connect to my server.

Specifically it will write this on my server:
net:180: bad argument: string expected, got nil

As far as I can tell it happens because the 'packet_receive' function returns nothing. But it IS getting the parameters it expects

Any idea what is going on?

#3 tirankenja

  • New Members
  • 4 posts

Posted 20 May 2012 - 10:24 AM

Ah. Nevermind. I replaced the api files and rebooted. And not it seems to be working

#4 tirankenja

  • New Members
  • 4 posts

Posted 20 May 2012 - 12:21 PM

And then it started failing again. Seems that it will crash because the routeTable is empty. So it never loops though it in net.packet_receive. But I am not quite sure why it sometimes work and sometimes not

#5 Imgoodisher

  • Members
  • 67 posts
  • LocationYes

Posted 20 May 2012 - 03:13 PM

I've been trying to make something with this, but its not working. for some reason the client won't send anything to the server. I'm pretty sure I have everything set up right, but its not very obvious how to set it up so I can't be sure I got everything. I have a client and server with
shell.run("modread")
shell.run("netd")
net.netInit()
in it, and a router with the same thing but with a shell.run("routed") before the net.netInit(). All the computers are labeled (server is labeled bankserver"), the server is running a daemon program that is pretty much the same thing as the one in the readme (name is account), but when I run connection.open("bankserver", "account", 2) it returns false. Why? did I miss something?

#6 tirankenja

  • New Members
  • 4 posts

Posted 20 May 2012 - 06:46 PM

Not getting a connection for me seems to either being the server reusing an unclosed connection (the server in the readme seems to do that). Or if the /etc/hosts file not having the server in it. Provoking the latter to happen seems a bit random to me so far. But once it is there it seems to be working.

Edit:
So far I have been unable to get a connection over a router. So that could also be the problem

#7 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 20 May 2012 - 11:38 PM

Okay, I will start a new world and attempt to reproduce these errors. Thanks for the feedback, I should have answers for you guys shortly!

#8 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 21 May 2012 - 12:00 AM

Unfortunately, I can't reproduce either of these on a fresh world; my simple three-computer network (one router, two clients) manages to set up fine.

I will say that routing tables have trouble populating if a router isn't the first computer set up, or if computers previously set up are not rebooted after the last one is set up. This is because only routers will automatically distribute their route table when a computer starts up, to avoid some serious lag issues in dense networks.

View PostImgoodisher, on 20 May 2012 - 03:13 PM, said:

I've been trying to make something with this, but its not working. for some reason the client won't send anything to the server. I'm pretty sure I have everything set up right, but its not very obvious how to set it up so I can't be sure I got everything. I have a client and server with
shell.run("modread")
shell.run("netd")
net.netInit()
in it, and a router with the same thing but with a shell.run("routed") before the net.netInit(). All the computers are labeled (server is labeled bankserver"), the server is running a daemon program that is pretty much the same thing as the one in the readme (name is account), but when I run connection.open("bankserver", "account", 2) it returns false. Why? did I miss something?

To help you troubleshoot this issue, I would like to see the route table (etc/hosts) on the router, the computer "bankserver" and the client attempting to connect to bankserver. Also, they are all inside rednet range, correct?

Here is a stripped-down network utility (programs/net) that allows you to view the route table or delete the route table (from memory and disk) on any computer:

local tArgs = { ... }
if tArgs[1] == "route" then
    --routes stuff
    if tArgs[2] == "print" then
        for rId, rInfo in pairs(net.routeTable) do
            print(rId..": "..rInfo.name.." ("..rInfo.gateway..":"..rInfo.cost..")")
        end
    elseif tArgs[2] == "flush" then
        net.routeTable = {}
        return net.netInit()
    end
else
    print("Usage: net route <print|flush>")
    --print("	   net shell <servername>")
end

View Posttirankenja, on 20 May 2012 - 06:46 PM, said:

Not getting a connection for me seems to either being the server reusing an unclosed connection (the server in the readme seems to do that). Or if the /etc/hosts file not having the server in it. Provoking the latter to happen seems a bit random to me so far. But once it is there it seems to be working.

Edit:
So far I have been unable to get a connection over a router. So that could also be the problem

Re-using unclosed connections generally hasn't caused any issues for me. I would be very interested to see it doing so. If there's anything you can send me to help reproduce the issue, I'd be very appreciative. Try the following in a new creative world (same as survival, but much faster for testing):

Create and label three computers. Set up one as a router and reboot it:

shell.run("modread")
shell.run("netd")
shell.run("routed")
net.netInit()

Then set up the other two as normal computers, and reboot them:

shell.run("modread")
shell.run("netd")
net.netInit()

If you have filed/filec installed, try running filed on one of the normal computers and running filec from the other. This should work. If so, try creating another computer just out of range of one normal computer, but in range of the router. Attempt a connection through the router this way.

To both; looking at the code, there is also an issue with an empty packet message. If a packet is sent with no message, it will break things due to the way packets are handled when being received. I'll look into adding a check for that, but the assumption that the packet message isn't empty is fairly common throughout the code.

#9 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 22 May 2012 - 12:49 AM

Updated with modified versions of rom/exit, rom/shutdown, and rom/reboot. Each of the modified versions simply adds a line at the top:

coroutine.resume(net.daemonTable.netd, 0, "SI:stop")

Which tells the network daemon (netd) to shut down.

#10 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 14 July 2012 - 08:36 AM

The LyqydNet API has been updated to use numbered sockets for input/output instead of server app names. Certain numbered sockets are reserved for certain types of programs, as follows:
  • 4 - network/routing daemons
  • 18 - relayed chat
  • 21 - file transfers
  • 23 - telnet/ssh/remote console
  • 25 - email
Sockets above 50 are used as-needed for outgoing connections, but could also be used to listen on if desired.

The new version also offers an unfinished version of terminal redirection over rednet. Using terminal.redirect(connection.text(#)) (replacing # with a connection number) will cause all screen commands to be sent over that connection. Use connection.processText() on the other end to receive it. Examples can be found in the LyqydNet-Programs repository (net and nshd). The net program, nshd, filec, and filed have all been updated and tested. These will work with the numbered sockets.

Proper documentation updates to follow at a later date.

#11 minizbot2012

  • Members
  • 122 posts
  • LocationPalm Bay, Florida

Posted 15 July 2012 - 11:21 PM

Would it be possible to do a net shell in a net shell? that would be amazing, but as it seems right now i cannot edit a new file in a net shell (bug?). but i can seem to edit an existing file.

#12 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 16 July 2012 - 02:26 AM

No, a shell session within another shell session isn't currently possible (which implies an area of insufficiency in handling programs that require input), so that's the next step for it. That's probably going to be a tricky one, so it may not be soon. However, any net shell server available at one will be available at the first, so you could (for now) simply connect directly to the second one.

As for the file editing, that is certainly a bug. I'll look into that as soon as I can.

#13 kazagistar

  • Members
  • 365 posts

Posted 16 July 2012 - 11:53 AM

1) Is it SMP safe? If you can verify the correct functioning of all routers you have installed, can you know that packets have not been tampered with or inserted?
2) Is setup manual only? When you place a computer, do you need to manually specify its connections?

#14 minizbot2012

  • Members
  • 122 posts
  • LocationPalm Bay, Florida

Posted 16 July 2012 - 08:32 PM

EDIT: this sounded stupid so i put it in spoilers
Spoiler


#15 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 17 July 2012 - 12:00 AM

View Postkazagistar, on 16 July 2012 - 11:53 AM, said:

1) Is it SMP safe? If you can verify the correct functioning of all routers you have installed, can you know that packets have not been tampered with or inserted?
2) Is setup manual only? When you place a computer, do you need to manually specify its connections?
  • To the extent that you can trust the endpoints and any routers in between, yes. If you don't trust both endpoints and any connecting routers, this is no more or less secure than any other networking system.
  • Setup is done automatically. Each router builds and maintains a routing table of hosts, which it passes to each new host that connects. Hosts within range of each other will automatically update their routes to newly connected hosts, but the newly connected host has no knowledge of this. Newly joined hosts' information is not automatically distributed to other computers, but any event that causes the computer to reboot (chunk unload/reload, etc) will cause the updates to propagate.
Excellent questions! I'd be happy to provide further details of the exact process used in various aspects of the networking protocol.

#16 kazagistar

  • Members
  • 365 posts

Posted 17 July 2012 - 12:14 AM

This sounds really interesting, but I have a few questions... A is connected to B, and C wants to connect, and is only in range of B.

1) How does B know that C is a trusted host? How does C know that B is a trusted network access?

2) When does A get informed that it is connected to C (or how does it route messages there)?

3) When does C get informed that it is connected to A (or how does it route messages there)?

4) If there are any race conditions for loops, how do you handle them?

5) How do you deal with finding shortest paths, or do you do this?

Or do I misunderstand what automatic configuration entails?

#17 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 17 July 2012 - 12:34 AM

View Postminizbot2012, on 15 July 2012 - 11:21 PM, said:

Would it be possible to do a net shell in a net shell? that would be amazing, but as it seems right now i cannot edit a new file in a net shell (bug?). but i can seem to edit an existing file.

The API has been updated (the netfile API, specifically) to address this bug. New files are now editable just like any other file. Thanks for the bug report!

View Postkazagistar, on 17 July 2012 - 12:14 AM, said:

This sounds really interesting, but I have a few questions... A is connected to B, and C wants to connect, and is only in range of B.

1) How does B know that C is a trusted host? How does C know that B is a trusted network access?

2) When does A get informed that it is connected to C (or how does it route messages there)?

3) When does C get informed that it is connected to A (or how does it route messages there)?

4) If there are any race conditions for loops, how do you handle them?

5) How do you deal with finding shortest paths, or do you do this?

Or do I misunderstand what automatic configuration entails?
  • Neither B nor C have any way of knowing that the other can be "trusted". Routers are trusted implicitly, so malevolent routers are definitely something to be concerned with. If you have any ideas on workable trust schemes, please let me know. I haven't had reason to worry about it yet. I'd put network infrastructure interference on the same level as griefing, though.
  • A will be informed the next time it is rebooted (manually, chunk reload, world reload) that A exists by B, assuming B is running the routed daemon. Alternatively, under the all-computers-are-routers scheme, it would know right away by B informing a fellow router that a new host has come online (this is not suggested due to network loads; all efforts to minimize routers should be made)
  • Again assuming that B is a router and already knows about A, C will be told by B as soon as it comes online that B exists, and that A is one hop away through B.
  • Race conditions are mostly avoided and/or ignored. All other things being equal, with two router-to-router notifications traveling around a loop, the first one to arrive would be considered the best. A later one with lower cost is even better.
  • Shortest paths are found fairly quickly through the gateway/cost system I use to keep track of routes. Each router and host only knows the current best way to get to any given host for the first hop. It knows that sending a packet to router X will get the packet to host Z, and that it will pass through n hops to get there. If a better route becomes known, it will be used instead. So when each host comes online, any routers in range will forward on its information to all other routers that the nearest routers know about. It sends this information directly to each router, rather than trying to propagate the information between routers (which leads to infinite loops and dragons). So basically, we discard worse routes as better ones become known (by comparing the number of hops to the host between the two). Also, any time a packet is directly received (cost zero), we immediately update the routing table if we had a non-zero cost previously.


#18 kazagistar

  • Members
  • 365 posts

Posted 17 July 2012 - 01:21 AM

I see, so your network is somewhat static, in that it does not really handle path removal? Or do you just ignore the potential race condition of sending a "remove" and "connect" at near the same time?

Why don't you just instantly propagate network architecture changes? Have each router send a message to each connected router upon connect? Or just have another layer of protocol to handle this in some way?

And as for security, I would use a whitelist + password. You can get a list of computers on a network from a networked computer, toss it on a floppy, and have the new computer only connect to those trusted computers. It sends to each on the list instead of broadcast to try to connect. It sends a password with its request, which is unique to the network. If any network computer gets compromised, it will fail, but we can't do much better then that, without full public key cryptography.

#19 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 17 July 2012 - 02:11 AM

View Postkazagistar, on 17 July 2012 - 01:21 AM, said:

I see, so your network is somewhat static, in that it does not really handle path removal? Or do you just ignore the potential race condition of sending a "remove" and "connect" at near the same time?

Why don't you just instantly propagate network architecture changes? Have each router send a message to each connected router upon connect? Or just have another layer of protocol to handle this in some way?

And as for security, I would use a whitelist + password. You can get a list of computers on a network from a networked computer, toss it on a floppy, and have the new computer only connect to those trusted computers. It sends to each on the list instead of broadcast to try to connect. It sends a password with its request, which is unique to the network. If any network computer gets compromised, it will fail, but we can't do much better then that, without full public key cryptography.

Instant updates to other routers upon host join is the current behavior; instant updates to hosts is not (and is this way to reduce network lag issues upon loading chunks with high computer density). Host removal is planned, but I haven't settled on the conditions for removing a host. Right now, this would be the sort of thing one would do manually at each router and only very carefully. The network is very good at not losing host information.

A whitelisted mode is certainly a possibility, but I doubt if it would become the default behavior. The network API is designed to very quickly get a fully-functional network set up and running. Add a couple lines to each host, pick a computer to be the router (it can work without one, but having a router right off the bat makes things easier, certainly) and you're off to the races. Having people set up whitelists goes against the already-works-out-of-the-box idea, so it would be a special configuration option (just like the long-link functionality in routers currently is). PKI would be nice, but I don't see it happening. It would certainly be resource-intensive to the point of being prohibitive on most setups.

Edit: I don't know if it would interest you, but there is currently an open issue on the LyqydNet github repository in regards to route invalidation.

#20 minizbot2012

  • Members
  • 122 posts
  • LocationPalm Bay, Florida

Posted 17 July 2012 - 08:45 PM

The example client / server pair is out of date. Is it possible that you can provide updated example client / servers?





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users