Jump to content




A Smaller Top Level Coroutine Override


38 replies to this topic

#21 McLeopold

  • Members
  • 123 posts

Posted 04 September 2013 - 05:20 PM

View Posttheoriginalbit, on 04 September 2013 - 03:50 PM, said:

View PostMcLeopold, on 04 September 2013 - 03:48 PM, said:

Okay, so what is not good about the top level?
Nothing is bad about the top level. The top level is good! It is fixing the bad thing!

I do believe that I said

View Posttheoriginalbit, on 02 September 2013 - 05:51 PM, said:

Ok so the common thing that people do with their Graphical Shells/OSes is to run it or a new shell from within the shell, as their files run in the startup; This is not good.
Note the ";" there, in English this indicates that the two sentences are related (there is more to it, but this is the basics).

I'm not asking what, I'm asking why. My first question was assuming that running an os in the shell was bad because it ran along side rednet. Let me step back an assumption and ask again.

As far as I can tell, your trading one call stack:

bios.lua:497 pcall, parallel.waitForAny
parallel:57  runUntilLimit
parallel:20  coroutine.resume
shell:192    shell.run
shell:32     os.run
bios.lua:365 pcall, fnFile

for another:

bios.lua:509 printError

with the additional that if you yield (or call os.pullEvent/pullEventRaw) the rednet code will queue an extra event for you sometimes.

So, what is bad about a shell (or high powered program) running inside of the main shell? What about the first call stack causes issues or slowdowns or whatever that the second call stack doesn't? What are the symptoms that I would see by not using this?

#22 theoriginalbit

    Semi-Professional ComputerCrafter

  • Moderators
  • 7,332 posts
  • LocationAustralia

Posted 04 September 2013 - 05:41 PM

View PostMcLeopold, on 04 September 2013 - 05:20 PM, said:

-snip-
There was a discussion about this in the original topic thread. I shall pull everything that jumps out at me and post them here.

Spoiler


#23 McLeopold

  • Members
  • 123 posts

Posted 04 September 2013 - 06:19 PM

View Posttheoriginalbit, on 04 September 2013 - 05:41 PM, said:

There was a discussion about this in the original topic thread. I shall pull everything that jumps out at me and post them here.

The discussion didn't talk about possible issues with running in a shell. The only problem mentioned was wanting to get rid of the rednet coroutine due to interference. All of the ideas could be implemented without the top level coroutine override. At this point it seems like colloquial wisdom that isn't grounded with any technical basis.

Quote

it allows you to run other functions/programs in parallel with the shell
Lua coroutines all run parallel regardless of which point in the call stack they were started. It isn't true threading, just "green" threads, which means they all share the same CPU/VM and must behave nicely by giving up control regularly. CC has something in the java layer that will kill a coroutine if it has gone too long without yielding, but this isn't from the bios.lua, so you are still under it's control.

Quote

it is perfect for a rednet networking system. every single wireless turtle/PC can perform its usual functions while also forwarding messages onwards
This is possible without the coroutine override. You can pull modem_message and relay messages just as well at any level.

Quote

if you use this injection you can easly replace shell functions and os functions without replacing stuff twice.
Shell and os functions can be replaced without the coroutine override.

Quote

This was actually for an Adhoc network api I was writing, and RedNet was getting in the way at the time, so I needed a way to get rid of it :P/>
This is the only reason I can see for doing it, but even then if the shell and rednet routine both get all messages, it shouldn't have interfered. I'd be interested to see what the interference actually was and if it was real or perceived.

Spoiler

The rest refer to getting rid of the shell so it isn't running in the "background". If you have a single startup file, the shell will give you control inside of a pcall until you error, so technically nothing else is "running in the background." If you end your program by exiting the shell, then no one can use the shell. So none of those reasons or ideas are really dependent on having an override. Filtering and manipulating events can be done by overriding os.pullEvent, again without the need for an override.

My question about the call stacks still stands. Any thoughts?

#24 theoriginalbit

    Semi-Professional ComputerCrafter

  • Moderators
  • 7,332 posts
  • LocationAustralia

Posted 04 September 2013 - 06:31 PM

View PostMcLeopold, on 04 September 2013 - 06:19 PM, said:

-snip-
Ok here is my response to you..... Don't use this override... You clearly cannot see any merits to anything people are saying, you have your own reasoning for why they are wrong and how this override makes no sense, you think it is fine to run a shell within a shell, so go ahead, you can run your shell in a shell, and the rest of us that use this will have our shells running as the parent shell with our routines as top level...


View PostMcLeopold, on 04 September 2013 - 06:19 PM, said:

My question about the call stacks still stands. Any thoughts?
Honestly I didn't understand the question. Makes no sense to me.

#25 McLeopold

  • Members
  • 123 posts

Posted 04 September 2013 - 08:46 PM

View Posttheoriginalbit, on 04 September 2013 - 06:31 PM, said:

Ok here is my response to you.....You clearly cannot see any merits ...you have your own reasoning for why they are wrong...you think it is fine to run a shell within a shell...and the rest of us...

Honestly I didn't understand the question. Makes no sense to me.

That last response wasn't the nicest, nor did it pertain to the question. We're on the same side, CC program enthusiasts.

I'm asking for a technical reason why a shell in a shell is bad. So far I've seen everyone take it on faith that it is without thinking about the technical implications to the lua vm.

Other than not wanting rednet to run, and maybe giving yourself room for a few extra recursive calls, I can't see any other reason why this would be better. Besides the very small instruction set that is in the first call stack, there is not much running. The only other gotchas I can think of would involve pcalls or coroutines and how they are specifically implemented in LuaJ or CC.

I'm looking for someone with the experience of running into CC issues or bugs that have seen the override fix their issues to reply. If you can't answer *that* question, please stop and let someone else who understands give the answer.

#26 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 05 September 2013 - 12:59 AM

The main use case is rednet networking APIs or other things where a coroutine blindly queuing rednet_message events dependent solely on which channels are open can foul things up. There's a feature for some of my stuff that's somewhat on the back burner right now that TLCO would solve perfectly if it was something I was willing to use in that program.

#27 McLeopold

  • Members
  • 123 posts

Posted 05 September 2013 - 10:46 AM

View PostLyqyd, on 05 September 2013 - 12:59 AM, said:

The main use case is rednet networking APIs or other things where a coroutine blindly queuing rednet_message events dependent solely on which channels are open can foul things up. There's a feature for some of my stuff that's somewhat on the back burner right now that TLCO would solve perfectly if it was something I was willing to use in that program.

Do you think we could come up with some example code that shows the issue and then a TLCO that fixes it? I would be interested in putting that use case together, but don't understand yet what the issue could be.

#28 Lyqyd

    Lua Liquidator

  • Moderators
  • 8,465 posts

Posted 05 September 2013 - 11:47 AM

Let's say you're using a networking API that uses rednet-style channels, but not the broadcast channel. You have another program running on the same machine that's using regular rednet. You've got some multitasking gizmo running these. Your networking API consumes its custom packets and queues rednet_message events under specific circumstances. In a single-program environment, the event queued by the custom API is the only rednet_message event that gets queued (because the custom API doesn't open the broadcast channel). Since we are multitasking, the other program opens up the broadcast channel, and the rednet coroutine starts queuing rednet_message events for the custom API's unprocessed input packets.

This is one example use case. Again, the only valid practical reason for TLCO is to work around the rednet coroutine. There are other legitimate reasons like, "Because I felt like it," or, "Because it seemed like fun," of course. The only practical reason is rednet though.

#29 Hawk777

  • Members
  • 162 posts

Posted 08 September 2013 - 12:47 PM

Why is this useful? How about because it cuts the number of (real-world) Java Minecraft threads in half or more, because every coroutine uses a thread? A decent OS will make sleeping threads not take up many resources, but eventually there is a limit after which you won’t be able to run any more computers.

#30 ElvishJerricco

  • Members
  • 803 posts

Posted 08 September 2013 - 12:52 PM

View PostHawk777, on 08 September 2013 - 12:47 PM, said:

Why is this useful? How about because it cuts the number of (real-world) Java Minecraft threads in half or more, because every coroutine uses a thread? A decent OS will make sleeping threads not take up many resources, but eventually there is a limit after which you won’t be able to run any more computers.

This is wrong. Coroutines aren't real threads. The LuaJ VM is just allowed to choose what Lua code is running at what time. If coroutines were multithreaded, we could run two of the simultaneously.

#31 Geforce Fan

  • Members
  • 846 posts
  • LocationMissouri, United States, America, Earth, Solar System, Milky Way, Universe 42B, Life Street, Multiverse, 4th Dimension

Posted 08 September 2013 - 02:45 PM

So wait, does this do the same job as NeverCast's without any bugs? I'm confused with all the people posting patches and the code staying the same.

#32 Hawk777

  • Members
  • 162 posts

Posted 08 September 2013 - 10:55 PM

View PostElvishJerricco, on 08 September 2013 - 12:52 PM, said:

View PostHawk777, on 08 September 2013 - 12:47 PM, said:

Why is this useful? How about because it cuts the number of (real-world) Java Minecraft threads in half or more, because every coroutine uses a thread? A decent OS will make sleeping threads not take up many resources, but eventually there is a limit after which you won’t be able to run any more computers.

This is wrong. Coroutines aren't real threads. The LuaJ VM is just allowed to choose what Lua code is running at what time. If coroutines were multithreaded, we could run two of the simultaneously.

Sorry, but it’s actually not wrong. Each LuaJ coroutine is implemented as a native Java thread. The reason why two of them can’t run simultaneously is simply because they’re written to block until that particular coroutine is supposed to run. If you don’t believe me, run Minecraft, create a bunch of coroutines, and then go check out how many system threads your Java process has (in Linux this is very easy to do, just check how many subdirectories there are in /proc/`pidof java`/task).

#33 ElvishJerricco

  • Members
  • 803 posts

Posted 09 September 2013 - 12:44 AM

View PostHawk777, on 08 September 2013 - 10:55 PM, said:

Sorry, but it’s actually not wrong. Each LuaJ coroutine is implemented as a native Java thread. The reason why two of them can’t run simultaneously is simply because they’re written to block until that particular coroutine is supposed to run. If you don’t believe me, run Minecraft, create a bunch of coroutines, and then go check out how many system threads your Java process has (in Linux this is very easy to do, just check how many subdirectories there are in /proc/`pidof java`/task).

I'm sorry. I was going off of my knowledge of Lua, and expecting LuaJ to be a straight port of Lua. It appears LuaJ does in fact do it differently. In my opinion, it is a mistake of LuaJ's to do coroutines in such a way and it could have been easily avoided by sticking closely to the C code.

#34 Hawk777

  • Members
  • 162 posts

Posted 09 September 2013 - 01:03 AM

View PostElvishJerricco, on 09 September 2013 - 12:44 AM, said:

I'm sorry. I was going off of my knowledge of Lua, and expecting LuaJ to be a straight port of Lua. It appears LuaJ does in fact do it differently. In my opinion, it is a mistake of LuaJ's to do coroutines in such a way and it could have been easily avoided by sticking closely to the C code.

I agree, it’s a bit odd. The reason LuaJ does this is that, if I’m not mistaken, LuaJ isn’t a pure interpreter; rather, it translates Lua code into JVM bytecode in dynamically generated classes and then runs it natively in the JVM. This has the problem that Lua’s execution state is now implemented as a Java execution state, and the lightest-weight thing Java provides that can encapsulate an execution state is a thread—Java doesn’t have coroutines, therefore LuaJ can’t implement coroutines as cheaply as you would expect while still translating to Java bytecode.

#35 ElvishJerricco

  • Members
  • 803 posts

Posted 09 September 2013 - 01:07 AM

View PostHawk777, on 09 September 2013 - 01:03 AM, said:

I agree, it’s a bit odd. The reason LuaJ does this is that, if I’m not mistaken, LuaJ isn’t a pure interpreter; rather, it translates Lua code into JVM bytecode in dynamically generated classes and then runs it natively in the JVM. This has the problem that Lua’s execution state is now implemented as a Java execution state, and the lightest-weight thing Java provides that can encapsulate an execution state is a thread—Java doesn’t have coroutines, therefore LuaJ can’t implement coroutines as cheaply as you would expect while still translating to Java bytecode.

If I'm not mistaken, the JVM is the fastest virtual machine out there except for maybe LuaJIT2 (for those that don't know, this is not Lua nor LuaJ). Both of them achieve near-C speeds, so I'm surprised they didn't feel comfortable doing an interpreter.

EDIT: Just realized that if we were using LuaJIT, we could write a Lua interpreter in Lua with decent performance... Meh.

#36 Hawk777

  • Members
  • 162 posts

Posted 09 September 2013 - 01:17 AM

View PostElvishJerricco, on 09 September 2013 - 01:07 AM, said:

If I'm not mistaken, the JVM is the fastest virtual machine out there except for maybe LuaJIT2 (for those that don't know, this is not Lua nor LuaJ). Both of them achieve near-C speeds, so I'm surprised they didn't feel comfortable doing an interpreter.

EDIT: Just realized that if we were using LuaJIT, we could write a Lua interpreter in Lua with decent performance... Meh.

Well, I don’t know. The JVM is pretty decent, so I think it makes perfect sense for LuaJ to generate JVM bytecode as output—if it’s even somewhat sensible, it means the Lua code gets to run really fast by taking advantage of all the JVM’s optimizations.

#37 ElvishJerricco

  • Members
  • 803 posts

Posted 09 September 2013 - 01:25 AM

View PostHawk777, on 09 September 2013 - 01:17 AM, said:

Well, I don’t know. The JVM is pretty decent, so I think it makes perfect sense for LuaJ to generate JVM bytecode as output—if it’s even somewhat sensible, it means the Lua code gets to run really fast by taking advantage of all the JVM’s optimizations.

According to the homepage (http://luaj.org/luaj.html), it is in fact an interpreter. It doesn't look like LuaJ is Lua→JVM Bytecode.

#38 Hawk777

  • Members
  • 162 posts

Posted 09 September 2013 - 01:28 AM

View PostElvishJerricco, on 09 September 2013 - 01:25 AM, said:

According to the homepage (http://luaj.org/luaj.html), it is in fact an interpreter. It doesn't look like LuaJ is Lua→JVM Bytecode.

Except for this sentence on that very page:

Quote

In this model the lua bytecode can be compiled directly into Java bytecode, lua source can be turned into Java source, lua bytecode can be interpreted, or all techniques can be mixed and matched depending the needs of the host runtime.

I guess it provides multiple options, but I think ComputerCraft uses JITing (and certainly the sentence “Version 2.0 is a significant rewrite whose implementation leverages the Java stack instead of a separate lua stack” explains why Java threads have to be used to implement coroutines).

#39 NeverCast

  • Members
  • 400 posts
  • LocationChristchurch, New Zealand

Posted 16 June 2019 - 12:17 PM

Well isn't this a far more elequent design :)





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users