99 replies to this topic

[Yevano]

SquidDev, on 05 May 2014 - 02:19 PM, said:

I just downloaded this and I am amazed. This is by far and away one of the best things I've discovered that is made for CC.

Just one thing I found bug (or problem) I found when messing around with it: When you override the toString method, you have to return a String object, rather than the default Lua string. Otherwise you get an error. I think this is caused by see\bin\see (line 138): return value:toString():lstr(). I don't know if this is intentional but my 'fix' is:

local str = value:toString()
if type(str) == "string" then
	return str
end
return str:lstr()

Otherwise this is such a great resource. Several things I would like to be seen in the future:

• Proper event manager: This would be mainly for GUIs. Ignore me. I just read through your WeeGUI code and you already have that implimented but it would be nice to see this in the core SEE libary.
  
• Sandboxing: I figure that if SEE has direct control over APIs then you could limit access to particular APIs. For instance ban access to websites or force a program to only run in one folder. This would be useful not only for untrusted code (not that there is much on CC) but also for debugging - you wouldn't be able to muck up the current file structure.
  
• Extensions to the current APIs: At the moment the SEE APIs (Rednet, Peripheral, etc...) simply replicate the built in APIs. It would be great if some new features could be added. This might include, for example, in the Rednet API one could make the side parameter for isOpen, open and close optional.

Thanks so much for creating this libary. If you want me to code anything I'm happy to help.

Glad you like it SquidDev! I don't work on this project much these days, though I do check the repo regularly for pull requests and such. If you want to work on some of those things you listed, you're definitely welcome to make some commits and send them my way!

toString should always return a see.base.String rather than a normal Lua string. It needs to do that because both user code and runtime code may call it. Since it can get called by user code directly, we don't want the user to have to worry about casting those strings. They should always be expecting a see.base.String. The line you're referencing is in see/apis/see, and it actually is for casting objects to normal strings.

About event handling, I'd like to keep the core event system pretty much the way it currently is. The main problem with listener style handling is that eventually you can get into callback hell with listeners inside listeners inside listeners (etc). That was actually the way I originally did the events system, using hooks (You can probably find them in some commits waaay back). The current events system works much better with the threading system, which by the way, you should totally play with that. One of the most fun parts of the code I think.

Of course, listener style event handling works pretty well for GUI systems because they're so object-oriented. That's why it has its own custom event system.

Now, sandboxing. I was thinking about doing that when someone was making that new browser that was supposed to be pretty awesome, but he ended up being our biggest critic so I didn't think too much more about it. Still though, SEE applets would be pretty damn cool for the new Firewolf I think. All you'd need to do is call the SEE API functions from Firewolf to load the runtime in. At that point, natives would have to be disabled and file APIs would need to be restricted. There could be something similar to Java like a SecurityManager or something to do the sandboxing. Maybe you get an instance of the SecurityManager and it tracks the level of the object. Code with access to higher level security managers would be able to sandbox potentially unsafe code by writing settings to the object. The same code would still be able to do as it pleases by temporarily disabling the manager when it needs to do some protected operations. Hmm, this might actually be a really neat idea.

SquidDev, on 07 May 2014 - 04:36 PM, said:

Issues with Rednet
I was playing around with the SEE Rednet API and I don't think it works. I am aware of the ModemInputStream and ModemOutputStream APIs but they are not as effective as the original Rednet API as they don't provide all functionality. I know I could just pull RendetMessageEvent (but you warn us against it) or ModemMessageEvent but this still does not give me the ability to use functions such as lookup.

I think the only bit of Rednet that does not work is receiving (so rednet.receive and rednet.lookup). I think what happens is that the "rednet_message" event is never found (as it is gobbled up by the /apis/see and queued internally instead). As a result neither of these two functions return.

This 'event gobbling' bug also affects other core functionality such as read (and System.read).
Short of re-creating the entire Rednet API I can't find a decent solution.

Sorry to sound so negative, everything else works perfectly.

I may ask sci4me about some of those things because he added in the rednet stuff for the update to CC1.6. It's quite possible he ended up just wrapping the functions directly instead of pulling the correct event. This a problem I had with sleep a while back, and then realized that I'd just need to queue and pull timer events manually in the code rather than calling the native sleep since the events system works differently.

Edit: Yep, he's directly calling rednet.receive and rednet.lookup. Those functions will end up pulling events the native way and just break stuff.

Edited by Yevano, 07 May 2014 - 09:19 PM.

[sci4me]

Yevano, on 07 May 2014 - 09:10 PM, said:

SquidDev, on 05 May 2014 - 02:19 PM, said:

I just downloaded this and I am amazed. This is by far and away one of the best things I've discovered that is made for CC.

Just one thing I found bug (or problem) I found when messing around with it: When you override the toString method, you have to return a String object, rather than the default Lua string. Otherwise you get an error. I think this is caused by see\bin\see (line 138): return value:toString():lstr(). I don't know if this is intentional but my 'fix' is:

local str = value:toString()
if type(str) == "string" then
	return str
end
return str:lstr()

Otherwise this is such a great resource. Several things I would like to be seen in the future:

• Proper event manager: This would be mainly for GUIs. Ignore me. I just read through your WeeGUI code and you already have that implimented but it would be nice to see this in the core SEE libary.
  
• Sandboxing: I figure that if SEE has direct control over APIs then you could limit access to particular APIs. For instance ban access to websites or force a program to only run in one folder. This would be useful not only for untrusted code (not that there is much on CC) but also for debugging - you wouldn't be able to muck up the current file structure.
  
• Extensions to the current APIs: At the moment the SEE APIs (Rednet, Peripheral, etc...) simply replicate the built in APIs. It would be great if some new features could be added. This might include, for example, in the Rednet API one could make the side parameter for isOpen, open and close optional.

Thanks so much for creating this libary. If you want me to code anything I'm happy to help.

Glad you like it SquidDev! I don't work on this project much these days, though I do check the repo regularly for pull requests and such. If you want to work on some of those things you listed, you're definitely welcome to make some commits and send them my way!

toString should always return a see.base.String rather than a normal Lua string. It needs to do that because both user code and runtime code may call it. Since it can get called by user code directly, we don't want the user to have to worry about casting those strings. They should always be expecting a see.base.String. The line you're referencing is in see/apis/see, and it actually is for casting objects to normal strings.

About event handling, I'd like to keep the core event system pretty much the way it currently is. The main problem with listener style handling is that eventually you can get into callback hell with listeners inside listeners inside listeners (etc). That was actually the way I originally did the events system, using hooks (You can probably find them in some commits waaay back). The current events system works much better with the threading system, which by the way, you should totally play with that. One of the most fun parts of the code I think.

Of course, listener style event handling works pretty well for GUI systems because they're so object-oriented. That's why it has its own custom event system.

Now, sandboxing. I was thinking about doing that when someone was making that new browser that was supposed to be pretty awesome, but he ended up being our biggest critic so I didn't think too much more about it. Still though, SEE applets would be pretty damn cool for the new Firewolf I think. All you'd need to do is call the SEE API functions from Firewolf to load the runtime in. At that point, natives would have to be disabled and file APIs would need to be restricted. There could be something similar to Java like a SecurityManager or something to do the sandboxing. Maybe you get an instance of the SecurityManager and it tracks the level of the object. Code with access to higher level security managers would be able to sandbox potentially unsafe code by writing settings to the object. The same code would still be able to do as it pleases by temporarily disabling the manager when it needs to do some protected operations. Hmm, this might actually be a really neat idea.

SquidDev, on 07 May 2014 - 04:36 PM, said:

Issues with Rednet
I was playing around with the SEE Rednet API and I don't think it works. I am aware of the ModemInputStream and ModemOutputStream APIs but they are not as effective as the original Rednet API as they don't provide all functionality. I know I could just pull RendetMessageEvent (but you warn us against it) or ModemMessageEvent but this still does not give me the ability to use functions such as lookup.

I think the only bit of Rednet that does not work is receiving (so rednet.receive and rednet.lookup). I think what happens is that the "rednet_message" event is never found (as it is gobbled up by the /apis/see and queued internally instead). As a result neither of these two functions return.

This 'event gobbling' bug also affects other core functionality such as read (and System.read).
Short of re-creating the entire Rednet API I can't find a decent solution.

Sorry to sound so negative, everything else works perfectly.

I may ask sci4me about some of those things because he added in the rednet stuff for the update to CC1.6. It's quite possible he ended up just wrapping the functions directly instead of pulling the correct event. This a problem I had with sleep a while back, and then realized that I'd just need to queue and pull timer events manually in the code rather than calling the native sleep since the events system works differently.

Edit: Yep, he's directly calling rednet.receive and rednet.lookup. Those functions will end up pulling events the native way and just break stuff.

Yes, I admit it. I did some silly things. We are now working on all the things now. I will go and fix the broken API's soon. We are currently working on a Security Manager system....... It sounds like it will be very interesting...

[SquidDev]

Yevano, on 07 May 2014 - 09:10 PM, said:

toString should always return a see.base.String rather than a normal Lua string. It needs to do that because both user code and runtime code may call it. Since it can get called by user code directly, we don't want the user to have to worry about casting those strings. They should always be expecting a see.base.String. The line you're referencing is in see/apis/see, and it actually is for casting objects to normal strings.

About event handling, I'd like to keep the core event system pretty much the way it currently is. The main problem with listener style handling is that eventually you can get into callback hell with listeners inside listeners inside listeners (etc). That was actually the way I originally did the events system, using hooks (You can probably find them in some commits waaay back). The current events system works much better with the threading system, which by the way, you should totally play with that. One of the most fun parts of the code I think.

Of course, listener style event handling works pretty well for GUI systems because they're so object-oriented. That's why it has its own custom event system.

All of that makes sense. I realised after posting that event listeners were only useful about 1 in 100 times as most of the time I have several 'event loops', making listeners less useful as you have to assign and unassign listeners to events.

Read

Quote

Why port over the entire function? You should just cast the parameters and pass them to the native function.

(from GitHub)
I felt I should explain more: read uses os.pullEvent (bios.lua:257) to work. It checks if it is a character, key, paste and performs an action based on that (such as entering a character). As SEE 'gobbles' up events from the os queue all read does is display a blinking cursor as it receives no events.

Rednet
I pushed a rednet fix to GitHub. Receive and lookup work.
I added a RednetConnection class. This allows you to send and receive rednet messages between two computers - a one-to-one connection. An array of these are returned on lookup. I chose to do this as lookup also gives you the ability to get the hostname, as well as filter by it so RednetConnection can store the hostname. Returning an array of these seemed better than returning an array of ids. I can always switch it back though if you think this isn't a great idea or there is a better way.

Rednet.receive returns the appropriate RednetMessageEvent. I wasn't sure about this but otherwise I would just end up creating another object that duplicates all data from RednetMessageEvent. There probably is a neater/better way, if so do tell me.

However the main issue I have is that rednet.host uses rednet.lookup (rednet:141). As host then sets tHostnames (which is a variable local to the rednet API), one cannot re-implement the functionality of it without porting the entire rednet loop which does not seem a not very good idea.

A side note: Why are we warned not to use RednetMessageEvent?

[Yevano]

Ah nice. Ok, I'll take a look at all this and probably merge it.

[Yevano]

SquidDev, on 09 May 2014 - 07:42 PM, said:

I felt I should explain more: read uses os.pullEvent (bios.lua:257) to work. It checks if it is a character, key, paste and performs an action based on that (such as entering a character). As SEE 'gobbles' up events from the os queue all read does is display a blinking cursor as it receives no events.

Gotcha. Merged.

SquidDev, on 09 May 2014 - 07:42 PM, said:

However the main issue I have is that rednet.host uses rednet.lookup (rednet:141). As host then sets tHostnames (which is a variable local to the rednet API), one cannot re-implement the functionality of it without porting the entire rednet loop which does not seem a not very good idea.

The simplest solution may be to call getfenv on rednet.host and detour its os.pullEvent so that it makes the correct calls. However, I'm not sure if a custom rednet event dispatcher would be a totally horrible thing. On the one side, we're of course basically duplicating old code and events and possibly making SEE's inner event loop have even more special cases, but on the other side, this would allow us to add our own custom parts to the rednet API, since it's not like rednet is doing anything magical anyways. I guess what I'm getting at is that we don't necessarily need to keep rednet as a standard, unless everyone thinks not keeping rednet as a standard is a really bad idea. A while ago I had started coding my own networking library using SEE which would do neat things like build route tables and do network discovery. It would also have been able to send out binary messages instead of just plaintext, using the current io system as the standard for doing such things. We'd be able to expand upon such a thing with connection-based sockets of course.

SquidDev, on 09 May 2014 - 07:42 PM, said:

A side note: Why are we warned not to use RednetMessageEvent?

I believe that warning has been there since before 1.6, when I basically deemed rednet as a small direct wrapper over modems. Now that rednet is more of a networking toolset, it makes more sense to use it, unless we were to make our own library for a similar purpose.

Edited by Yevano, 09 May 2014 - 09:00 PM.

[SquidDev]

Yevano, on 09 May 2014 - 08:58 PM, said:

The simplest solution may be to call getfenv on rednet.host and detour its os.pullEvent so that it makes the correct calls. However, I'm not sure if a custom rednet event dispatcher would be a totally horrible thing. On the one side, we're of course basically duplicating old code and events and possibly making SEE's inner event loop have even more special cases, but on the other side, this would allow us to add our own custom parts to the rednet API, since it's not like rednet is doing anything magical anyways. I guess what I'm getting at is that we don't necessarily need to keep rednet as a standard, unless everyone thinks not keeping rednet as a standard is a really bad idea. A while ago I had started coding my own networking library using SEE which would do neat things like build route tables and do network discovery. It would also have been able to send out binary messages instead of just plaintext, using the current io system as the standard for doing such things. We'd be able to expand upon such a thing with connection-based sockets of course.

Sorry. You lost me there. So I think what you are suggesting is that either we can recreate the rednet loop or use getfenv to redirect function calls. I quickly implemented the latter. I don't know if this is what you meant but it seemed simplest.

local hostEnv = getfenv(rednet.host)
hostEnv.lookup = Rednet.lookup

local status, err = pcall(rednet.host, protocol, name)

hostEnv.lookup = rednet.lookup

if not status then
    throw(Exception:new(err))
end

It simply redirects the rednet.lookup function to the SEE Rednet.looup, then returns it again. I have to use pcall to prevent rednet.host breaking and never getting its lookup back. Maybe this isn't the best solution?

[sci4me]

SquidDev, on 10 May 2014 - 08:40 AM, said:

Yevano, on 09 May 2014 - 08:58 PM, said:

The simplest solution may be to call getfenv on rednet.host and detour its os.pullEvent so that it makes the correct calls. However, I'm not sure if a custom rednet event dispatcher would be a totally horrible thing. On the one side, we're of course basically duplicating old code and events and possibly making SEE's inner event loop have even more special cases, but on the other side, this would allow us to add our own custom parts to the rednet API, since it's not like rednet is doing anything magical anyways. I guess what I'm getting at is that we don't necessarily need to keep rednet as a standard, unless everyone thinks not keeping rednet as a standard is a really bad idea. A while ago I had started coding my own networking library using SEE which would do neat things like build route tables and do network discovery. It would also have been able to send out binary messages instead of just plaintext, using the current io system as the standard for doing such things. We'd be able to expand upon such a thing with connection-based sockets of course.

Sorry. You lost me there. So I think what you are suggesting is that either we can recreate the rednet loop or use getfenv to redirect function calls. I quickly implemented the latter. I don't know if this is what you meant but it seemed simplest.

local hostEnv = getfenv(rednet.host)
hostEnv.lookup = Rednet.lookup

local status, err = pcall(rednet.host, protocol, name)

hostEnv.lookup = rednet.lookup

if not status then
	throw(Exception:new(err))
end

It simply redirects the rednet.lookup function to the SEE Rednet.looup, then returns it again. I have to use pcall to prevent rednet.host breaking and never getting its lookup back. Maybe this isn't the best solution?

Talking to yevano, I think we're going to re-write rednet in the see library.

[SquidDev]

sci4me, on 10 May 2014 - 09:16 AM, said:

Talking to yevano, I think we're going to re-write rednet in the see library.

I'm happy to code this. You guys seem to be making good progress with the Security manager. Just wondering how rednet will work though. I'm presuming Rednet.__static will spawn in a thread running an equivalent to rednet.run. Do we want compatibility with redet though?

Installer
I created an installer for SEE that pulls all files that aren't on a blacklist. I saw the GitHub issue and wrote a quick implementation. You'll probably want to change large portions of it though - it was more of a proof of concept.

Edited by SquidDev, 10 May 2014 - 11:34 AM.

[Yevano]

SquidDev, on 10 May 2014 - 10:51 AM, said:

sci4me, on 10 May 2014 - 09:16 AM, said:

Talking to yevano, I think we're going to re-write rednet in the see library.

I'm happy to code this. You guys seem to be making good progress with the Security manager. Just wondering how rednet will work though. I'm presuming Rednet.__static will spawn in a thread running an equivalent to rednet.run. Do we want compatibility with redet though?

I still have some old code that I can send you to give you an idea of how it was supposed to work. I'll make a new branch later today and you can pull it if you want. And yes, we should keep a separate library for keeping compatibility with rednet. If that code you wrote for lookup and receive works, then that should be all we need on that front I think.

SquidDev, on 10 May 2014 - 10:51 AM, said:

Installer
I created an installer for SEE that pulls all files that aren't on a blacklist. I saw the GitHub issue and wrote a quick implementation. You'll probably want to change large portions of it though - it was more of a proof of concept.

Awesome! That's something we've needed for a while. The only problem might be the calls to the github API. Apparently a non-authenticated user can only make 60 calls/hour. I'm thinking that should be enough for someone just installing every once in a while, but I could be mistaken on how many calls are actually made.

https://developer.gi.../#rate-limiting

EDIT: Here's the new branch for the net code I had worked on a while ago. https://github.com/Y...ee/tree/linknet
Please note it was being written a few months ago before we even had the new class system, but I did go ahead and fix any easily noticeable weirdness. Just take a look at it and see what you can gather of how it works. I'll explain more in depth soon when I get some time to myself.

Edited by Yevano, 10 May 2014 - 02:48 PM.

[SquidDev]

Yevano, on 10 May 2014 - 02:33 PM, said:

I still have some old code that I can send you to give you an idea of how it was supposed to work. I'll make a new branch later today and you can pull it if you want. And yes, we should keep a separate library for keeping compatibility with rednet. If that code you wrote for lookup and receive works, then that should be all we need on that front I think.

Great. I'll have a look. Thanks.

Yevano, on 10 May 2014 - 02:33 PM, said:

Awesome! That's something we've needed for a while. The only problem might be the calls to the github API. Apparently a non-authenticated user can only make 60 calls/hour. I'm thinking that should be enough for someone just installing every once in a while, but I could be mistaken on how many calls are actually made.

https://developer.gi.../#rate-limiting

The code only makes two requests - one for the sha hash, the other for the tree. Everything else is done through raw.github.com. I don't think the API call limit will apply as the calls are coming from different IPs. I might be wrong though.

Edited by SquidDev, 10 May 2014 - 02:59 PM.

[sci4me]

SquidDev, on 10 May 2014 - 02:58 PM, said:

Yevano, on 10 May 2014 - 02:33 PM, said:

I still have some old code that I can send you to give you an idea of how it was supposed to work. I'll make a new branch later today and you can pull it if you want. And yes, we should keep a separate library for keeping compatibility with rednet. If that code you wrote for lookup and receive works, then that should be all we need on that front I think.

Great. I'll have a look. Thanks.

Yevano, on 10 May 2014 - 02:33 PM, said:

Awesome! That's something we've needed for a while. The only problem might be the calls to the github API. Apparently a non-authenticated user can only make 60 calls/hour. I'm thinking that should be enough for someone just installing every once in a while, but I could be mistaken on how many calls are actually made.

https://developer.gi.../#rate-limiting

The code only makes two requests - one for the sha hash, the other for the tree. Everything else is done through raw.github.com. I don't think the API call limit will apply as the calls are coming from different IPs. I might be wrong though.

Nice work on the installer, I was working on that a while ago but the request limiting pissed me off too much to finish it lol.

[SquidDev]

Linknet
I'm slightly confused about this. I understand all the code it is just that I don't understand why. Why as in "Why does it do that?"; Some bits I can see as being really useful (Network discoverer) others I feel are just duplicating previous Rednet functionality.

[sci4me]

SquidDev, on 10 May 2014 - 09:32 PM, said:

Linknet
I'm slightly confused about this. I understand all the code it is just that I don't understand why. Why as in "Why does it do that?"; Some bits I can see as being really useful (Network discoverer) others I feel are just duplicating previous Rednet functionality.

I dont think its finished yet, but i'm sure Yevano will have better explanations. I think the idea is that it is intended to be a bit higher level than rednet...

[SquidDev]

For some strange reason I chose to write some serialization code. textutils.serialize works but textutils.unserialize just gives you a standard table. It works in the same way but for two major changes:

• The class name is stored in the __class property. This is used to find the required class.
  
• You can define custom serialization and deserialization functions. These should store a table or properties required to recreate the Object. For instance String returns {['lstr']='Your string']} instead of an array. The idea is that you can save space by not dumping everything.

This was more an experiment than useful or decent code some some bits may not be perfect.

[Yevano]

SquidDev, on 10 May 2014 - 09:32 PM, said:

Linknet
I'm slightly confused about this. I understand all the code it is just that I don't understand why. Why as in "Why does it do that?"; Some bits I can see as being really useful (Network discoverer) others I feel are just duplicating previous Rednet functionality.

Linknet is supposed to be a lower level yet more flexible version of Rednet. It's really supposed to model a bit closer to real life networking; a much simplified version of IP perhaps. Rednet is more high level with lots of abstractions, like its way of doing hostnames using broadcasting. With Linknet, messages aren't just broadcast around everywhere. The network discoverer need only broadcast itself every so often to its peers, and then when you need to send something to a remote computer, a check is done as to whether the address is mapped to a route yet. If not, the network discoverer hasn't found it yet. If yes, then the message is passed on to the next computer in the route. This continues until the message gets to the target computer. This is nice because it means that for large stable networks, all the computers aren't getting pinged every time a message is sent. Linknet might also have ports, that way we can do port blocking and such.

SquidDev, on 11 May 2014 - 06:52 PM, said:

For some strange reason I chose to write some serialization code. textutils.serialize works but textutils.unserialize just gives you a standard table. It works in the same way but for two major changes:

• The class name is stored in the __class property. This is used to find the required class.
  
• You can define custom serialization and deserialization functions. These should store a table or properties required to recreate the Object. For instance String returns {['lstr']='Your string']} instead of an array. The idea is that you can save space by not dumping everything.

This was more an experiment than useful or decent code some some bits may not be perfect.

Hmm, we should probably be doing binary serialization, though. Maybe have a class called Serializable with methods read(InputStream) and write(OutputStream). In the end, we could still do automatic serialization I think, too.

Edited by Yevano, 11 May 2014 - 08:24 PM.

[SquidDev]

Yevano, on 11 May 2014 - 08:24 PM, said:

Linknet is supposed to be a lower level yet more flexible version of Rednet. It's really supposed to model a bit closer to real life networking; a much simplified version of IP perhaps. Rednet is more high level with lots of abstractions, like its way of doing hostnames using broadcasting. With Linknet, messages aren't just broadcast around everywhere. The network discoverer need only broadcast itself every so often to its peers, and then when you need to send something to a remote computer, a check is done as to whether the address is mapped to a route yet. If not, the network discoverer hasn't found it yet. If yes, then the message is passed on to the next computer in the route. This continues until the message gets to the target computer. This is nice because it means that for large stable networks, all the computers aren't getting pinged every time a message is sent. Linknet might also have ports, that way we can do port blocking and such.

Ah. That makes far more sense. It sounds like it could be pretty cool once it all works.

Yevano, on 11 May 2014 - 08:24 PM, said:

Hmm, we should probably be doing binary serialization, though. Maybe have a class called Serializable with methods read(InputStream) and write(OutputStream). In the end, we could still do automatic serialization I think, too.

I wrote it like that simply to keep backwards compatibility with textutils.serialize and also if I wanted to implement other serialization (JSON, XML, etc...) it would be simple. Binary serialization would be interesting. You could use string.dump to even serialize functions! The Lua Wiki has some interesting ways to achieve this,

[Yevano]

SquidDev, on 12 May 2014 - 06:03 PM, said:

I wrote it like that simply to keep backwards compatibility with textutils.serialize and also if I wanted to implement other serialization (JSON, XML, etc...) it would be simple. Binary serialization would be interesting. You could use string.dump to even serialize functions! The Lua Wiki has some interesting ways to achieve this,

I have a pure Lua serializer and deserializer that I wrote a few months ago which supports all Lua types except for threads and other userdata. It supports functions and is also able to serialize self-referencing tables. On top of all that, the serialization is all binary. Maybe it will be of some use as a reference to you. https://gist.github....35cda144a9f9667

[SquidDev]

Yevano, on 13 May 2014 - 08:49 PM, said:

I have a pure Lua serializer and deserializer that I wrote a few months ago which supports all Lua types except for threads and other userdata. It supports functions and is also able to serialize self-referencing tables. On top of all that, the serialization is all binary. Maybe it will be of some use as a reference to you. https://gist.github....35cda144a9f9667

Thanks! I added a 'Binary' (de)serializer to my serialize branch.

Little feature request: Could @native statements have an optional as <asName> parameter. This would act a bit like Python's import statement does: [email protected] string.char as strchar would set strchr = string.char instead of importing it directly to string.char.

[Yevano]

SquidDev, on 16 May 2014 - 07:57 PM, said:

Yevano, on 13 May 2014 - 08:49 PM, said:

I have a pure Lua serializer and deserializer that I wrote a few months ago which supports all Lua types except for threads and other userdata. It supports functions and is also able to serialize self-referencing tables. On top of all that, the serialization is all binary. Maybe it will be of some use as a reference to you. https://gist.github....35cda144a9f9667

Thanks! I added a 'Binary' (de)serializer to my serialize branch.

Little feature request: Could @native statements have an optional as <asName> parameter. This would act a bit like Python's import statement does: [email protected] string.char as strchar would set strchr = string.char instead of importing it directly to string.char.

Sure. We should do that for imports as well. That would actually be a solution to the same-named imports problem.

Edit: https://github.com/Y...o/see/issues/47

Looked at your serialization/deserialization stuff. Could you give a quick run-down of how it all works? Btw, I noticed your abstract classes in see.io.serialization are explicitly throwing exceptions instead of using [email protected].

Edited by Yevano, 17 May 2014 - 03:26 PM.

[SquidDev]

Yevano, on 17 May 2014 - 03:16 PM, said:

Btw, I noticed your abstract classes in see.io.serialization are explicitly throwing exceptions instead of using [email protected]

Fixed. Sorry, I got confused by the use of the abstract 'keyword' and so didn't use it.

Yevano, on 17 May 2014 - 03:16 PM, said:

Looked at your serialization/deserialization stuff. Could you give a quick run-down of how it all works?

I'll have a go. Most of this is based on the .Net way of (de)serializing data. A good description of how that works can be found here.

Serialization
A simple serializer should look like:

[email protected] see.io.serialization.AbstractSerializer

function SimpleSerializer.serializeObject(object)
  if isprimitive(object) then
	-- Serialize simple objects: string, number, boolean, table and nil
  else
	-- Serialize objects: get the 'objct'
	local toSerialize = object:serialize()

	-- Give it a 'class' property (should copy the table to avoid overwriting but this is just an example)
	toSerialize['__class'] = object:getClass():getName():lstr()

	-- Serialize it as a table
	return SimpleSerializer.serializeObject(toSerialize)
  end
end

This is a simple serializer. To serialize an object it simply calls the object:serialize() function. This is defined in the Object class:

function Object:serialize()
  return self
end

However it can be overridden by other classes. This is mostly done if the class stores data which could be stored more simply. The String class stores its serialized data as a string rather than an Array:

function String:serialize()
  return {['lstr'] = self:lstr()}
end

This should always return a table. We set this table with a __class property with the class name in order to store what class this is.

Deserialization
A simple deserializer would work like:

[email protected] see.io.serialization.AbstractDeserializer
function SimpleDeserializer.deserializeObject(string)
  --Deserialize data, stepping through it. If we find a table:
  if objectType == "table"
	--Get class of object
	local class = object['__class']
	-- If this is an object then clear class variable
	if class then
	  object['__class'] = nil
	  -- Get the class from the string and deserialize it
	  object = __rt:loadClassFromAny(class):deserialize(object)
	end
  end
end

In actual fact, it should 'parse inside' the object before creating the object. The Class:deserialize method is slightly more complex then serialization:

function Class:deserialize(data)
  --If over-ride function exists then call it
  local override = __rt.classTables[self]["deserialize"]
  if override ~= nil then
	return override(data)
  end
  --Else create new class
  local instance = setmetatable({ __class = self }, __rt.classMT)
  for k,v in pairs(data) do
	instance[k] = v
  end
  return instance
end

What this does is check if the class contains a 'deserialize' method. If so, then it will call that instead. Otherwise we just create a new class and copy each piece of data into the created object. Custom deserialization would look something like:

function String.deserialize(data)
  return String:new(data['lstr'])
end

Simply creating a new string class with the lua string from the serialized data.

I hope that all made sense.

Edited by SquidDev, 18 May 2014 - 02:44 PM.