←  Bugs

ComputerCraft | Programmable Computers for Minecraft

»

[1.45] Rednet.send modifies any bytes > 127

Locked

devast8a's Photo devast8a 24 Oct 2012

Description of Bugs:
rednet modifies any bytes with a value > 127

Posted Image

First line being 01 -> 16
Every other line being hex representation of the byte at that position in the message.

Steps to Reproduce Bug:
rednet.send(id, string.char(n)) where n > 127

Client code, program has only one argument the id of the computer to send the data to.
local l = ""
for n,side in pairs(rs.getSides()) do
	if peripheral.getType(side) == "modem" then
		rednet.open(side)
		open = true
		break
	end
end

for i = 127 - 15, 127 + 16 * 4 do
	l = l .. string.char(i)
end
local args = {...}
rednet.send(tonumber(args[1]), l)
function memdmp(data)
	write("01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16\n")
  
	for i = 1, #data do
		write(string.format("%.2x ", string.byte(data:sub(i,i))))
	  
		if i % 16 == 0 then
			print()
		end
	  
		if i % 256 == 0 then
			print("Press any key to continue")
			os.pullEvent()
			write("01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16\n")
		end
	end  
	print()
end
memdmp(l)

Server code
for n,side in pairs(rs.getSides()) do
	if peripheral.getType(side) == "modem" then
		rednet.open(side)
		open = true
		break
	end
end
local event, p1, p2 = os.pullEvent()
function memdmp(data)
	write("01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16\n")
  
	for i = 1, #data do
		write(string.format("%.2x ", string.byte(data:sub(i,i))))
			  
		if i % 16 == 0 then
			print()
		end
	  
		if i % 256 == 0 then
			print("Press any key to continue")
			os.pullEvent()
			write("01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16\n")
		end
	end  
	print()
end
memdmp(p2)
Quote

faubiguy's Photo faubiguy 24 Oct 2012

I've successfully reproduced this bug in singleplayer 1.45. string.char(155) on the sending end was received as the character with byte 239. string.char(137) also resulted in the byte 239.
Quote

ElvishJerricco's Photo ElvishJerricco 24 Oct 2012

Actually not a bug with CC. Not a bug at all in fact. ASCII only goes up to 127. So converting a number to an ASCII symbol then back will lose anything over 127 because the number will be encoded into the ASCII character represented by 0 because it's unknown, which will be decoded to 0. Try sending the data in a table.
t = {num1, num2, num3 ... }
rednet.send(id, textutils.serialize(t))


id, ts = rednet.receive()
t = textutils.unserialize(ts)
Quote

devast8a's Photo devast8a 24 Oct 2012

It's not the conversion that's the problem.
You can convert to a symbol above 127 and back perfectly fine.

It's specifically a problem with rednet.send itself.
Quote

Cloudy's Photo Cloudy 24 Oct 2012

I'll look into it - but we don't do anything to the string which is passed java side. We merely pass it as an argument to the other modems receive function. So it looks like something is being lost in conversion from Lua to Java - which may be beyond our control.
Quote

Sebra's Photo Sebra 24 Oct 2012

View PostElvishJerricco, on 24 October 2012 - 06:18 AM, said:

Actually not a bug with CC. Not a bug at all in fact. ASCII only goes up to 127. So converting a number to an ASCII symbol then back will lose anything over 127 because the number will be encoded into the ASCII character represented by 0 because it's unknown, which will be decoded to 0. Try sending the data in a table.
Bug it or not this is bad. String message can be not a 7bit ASCII only. It can be not even 8bit letters. It is just a string value which should be delivered unchanged.
Sending chars as a table is like a byte reading/writing. I think binary io in CC is so bad exactly because of that ASCII recoding.

View PostCloudy, on 24 October 2012 - 08:45 AM, said:

I'll look into it - but we don't do anything to the string which is passed java side. We merely pass it as an argument to the other modems receive function. So it looks like something is being lost in conversion from Lua to Java - which may be beyond our control.
It seems Java count it as text and apply recoding ^_^/> Thanks for your efforts!
Quote

ElvishJerricco's Photo ElvishJerricco 24 Oct 2012

View PostSebra, on 24 October 2012 - 03:01 PM, said:

View PostElvishJerricco, on 24 October 2012 - 06:18 AM, said:

Actually not a bug with CC. Not a bug at all in fact. ASCII only goes up to 127. So converting a number to an ASCII symbol then back will lose anything over 127 because the number will be encoded into the ASCII character represented by 0 because it's unknown, which will be decoded to 0. Try sending the data in a table.
Bug it or not this is bad. String message can be not a 7bit ASCII only. It can be not even 8bit letters. It is just a string value which should be delivered unchanged.
Sending chars as a table is like a byte reading/writing. I think binary io in CC is so bad exactly because of that ASCII recoding.

View PostCloudy, on 24 October 2012 - 08:45 AM, said:

I'll look into it - but we don't do anything to the string which is passed java side. We merely pass it as an argument to the other modems receive function. So it looks like something is being lost in conversion from Lua to Java - which may be beyond our control.
It seems Java count it as text and apply recoding ^_^/> Thanks for your efforts!

You're really not understanding dude. It's like you have a table with 5 values, and you try to use the sixth, so it defaults and gives you 0. That's what's happening. ASCII has a different way of representing numbers. To create a char from the number 65 would give you "A", not "65". So don't expect ASCII to just be a bus on which your numbers can ride.

You're trying to use that beyond-the-scope value. The ASCII letters are still using the full byte, but only values up to 127 are values that can be made sense of in the context of characters. I assume this might be because ASCII was developed with an 8 bit signed integer in mind. Signed integers use 7 bits for value and one for whether it's positive or negative. So again, it defaults to 0 because there is no character for a value greater than 127. What are you wanting it to do? Spit out a non-existant character with your value just so that you can avoid using smarter methods of sending? So no. It's not a bad thing that this is happening. This is exactly what should be happening and it makes perfect sense.

And before you go on about how strings can't be just 7 or 8 bit ASCII values, let me inform you that they're not. Strings in low level languages are arrays of ASCII characters. And strings in high level languages like lua are objects.
Quote

faubiguy's Photo faubiguy 24 Oct 2012

I don't see a reason for rednet.send to send strings in the context of ASCII. A string in lua is a series of bytes, and doesn't necessarily store text, ASCII or otherwise. A string can be used to store a sequence of arbitrary bytes. However, as mentioned above, this is likely unintended result of conversion to java and back, and can't be easily fixed.
Quote

ElvishJerricco's Photo ElvishJerricco 25 Oct 2012

View Postfaubiguy, on 24 October 2012 - 07:17 PM, said:

I don't see a reason for rednet.send to send strings in the context of ASCII. A string in lua is a series of bytes, and doesn't necessarily store text, ASCII or otherwise. A string can be used to store a sequence of arbitrary bytes. However, as mentioned above, this is likely unintended result of conversion to java and back, and can't be easily fixed.

It's because he's using string.char() and string.byte(). Those functions' purpose are to convert a number to its ASCII character and back. Like i said, strings in lua are objects, so it's not the string's fault. It's the method used to go from number to string. The purpose is to use the ASCII standard which really should behave the way it is here. It is not a bug, it is not bad behavior, it is not Java's fault. It is exactly what it was designed to be.
Quote

ElvishJerricco's Photo ElvishJerricco 25 Oct 2012

View PostCloudy, on 24 October 2012 - 08:45 AM, said:

I'll look into it - but we don't do anything to the string which is passed java side. We merely pass it as an argument to the other modems receive function. So it looks like something is being lost in conversion from Lua to Java - which may be beyond our control.

Cloudy are you understanding what i'm trying to say? Because no one else seems to get it and the reason this is happening is mildly technical. I don't want the devs of the mod thinking that this is a problem that should be fixed.
Quote

BigSHinyToys's Photo BigSHinyToys 25 Oct 2012

View PostElvishJerricco, on 25 October 2012 - 03:33 AM, said:

View Postfaubiguy, on 24 October 2012 - 07:17 PM, said:

I don't see a reason for rednet.send to send strings in the context of ASCII. A string in lua is a series of bytes, and doesn't necessarily store text, ASCII or otherwise. A string can be used to store a sequence of arbitrary bytes. However, as mentioned above, this is likely unintended result of conversion to java and back, and can't be easily fixed.

It's because he's using string.char() and string.byte(). Those functions' purpose are to convert a number to its ASCII character and back. Like i said, strings in lua are objects, so it's not the string's fault. It's the method used to go from number to string. The purpose is to use the ASCII standard which really should behave the way it is here. It is not a bug, it is not bad behavior, it is not Java's fault. It is exactly what it was designed to be.

you are incorrect those functions work perfectly fine with binary example with the bellow code it is possible to load a file example a MIDI into a string and save it again causing no detrimental effects to the file it can be played after and it is not a ascii file.
Spoiler
Quote

ElvishJerricco's Photo ElvishJerricco 25 Oct 2012

View PostBigSHinyToys, on 25 October 2012 - 03:45 AM, said:

View PostElvishJerricco, on 25 October 2012 - 03:33 AM, said:

View Postfaubiguy, on 24 October 2012 - 07:17 PM, said:

I don't see a reason for rednet.send to send strings in the context of ASCII. A string in lua is a series of bytes, and doesn't necessarily store text, ASCII or otherwise. A string can be used to store a sequence of arbitrary bytes. However, as mentioned above, this is likely unintended result of conversion to java and back, and can't be easily fixed.

It's because he's using string.char() and string.byte(). Those functions' purpose are to convert a number to its ASCII character and back. Like i said, strings in lua are objects, so it's not the string's fault. It's the method used to go from number to string. The purpose is to use the ASCII standard which really should behave the way it is here. It is not a bug, it is not bad behavior, it is not Java's fault. It is exactly what it was designed to be.

you are incorrect those functions work perfectly fine with binary example with the bellow code it is possible to load a file example a MIDI into a string and save it again causing no detrimental effects to the file it can be played after and it is not a ascii file.
Spoiler

That's because you're dealing with numbers meant to be represented in ASCII! The number for an A in MIDI is 65 is it not? Not-so-coincidentally, that's the ASCII value for "A". That's exactly the intention. There simply is no ASCII character for values over 127. That's why they default to the ASCII character that 0 represents when you use string.char(numberGreaterThan127). So when you decode with string.byte, it gives 0. That. Is. The. Intention.
Quote

ElvishJerricco's Photo ElvishJerricco 25 Oct 2012

Ok I just did some testing and these are the results:

Posted Image

As you can see, characters over 255 (the highest eight bit value possible) create an error when you try to create a char from them. Anything less than that will be decoded using string.byte just fine... Locally that is. So why isn't it working over rednet? I'm still not sure... However, i was still sort of right =P Not in the important way though. I'll do more testing...
Quote

BigSHinyToys's Photo BigSHinyToys 25 Oct 2012

there s 256 values ranging from 0 to 255 so 256 will always fail as it is above that. also MIDI files are not text in nature
this is what one looks like when opened by Notepad ++
Spoiler
Quote

ElvishJerricco's Photo ElvishJerricco 25 Oct 2012

View PostBigSHinyToys, on 25 October 2012 - 04:13 AM, said:

there s 256 values ranging from 0 to 255 so 256 will always fail as it is above that. also MIDI files are not text in nature
this is what one looks like when opened by Notepad ++
Spoiler

Yea after a decent amount of testing, I don't really understand what's happening anymore. Apologies for being wrongly confident in false facts this whole time.
Quote

dan200's Photo dan200 25 Oct 2012

Is this rednet over bundled cable or rednet over modems?
Quote

dan200's Photo dan200 25 Oct 2012

Rednet isn't for sending "bytes", it's for sending strings of text. if you're trying to use strings to encode binary data using string.char/string.byte, you're making a lot of assumptions about character sets that may not hold true. If you really -must- send data in binary (i can't think why you would, sending the lua data you want using textutils.serialise, or just strings of numbers, is much simpler), write a base64 encoder.
Quote

Sebra's Photo Sebra 25 Oct 2012

View Postdan200, on 25 October 2012 - 02:21 PM, said:

Is this rednet over bundled cable or rednet over modems?
I got it over modems. Over cable it is bad too.
Not all languages fit in 7bit ASCII. Function textutils.serialize() do not hides upper half of bytes. There are strings in lua happens to be with bytes >127. Sometimes people want to send whole file as a rednet message. Yes, it is possible to recode it byte by byte, as in binary input/output, but that is quite inoptimal.
If ASCII recoding can be avoided please do it. If it cannot, we will use bad ways around it.
I just want a string sent and received to be the same.
Quote

BigSHinyToys's Photo BigSHinyToys 25 Oct 2012

View Postdan200, on 25 October 2012 - 02:23 PM, said:

Rednet isn't for sending "bytes", it's for sending strings of text. if you're trying to use strings to encode binary data using string.char/string.byte, you're making a lot of assumptions about character sets that may not hold true. If you really -must- send data in binary (i can't think why you would, sending the lua data you want using textutils.serialise, or just strings of numbers, is much simpler), write a base64 encoder.

base64 HEX or actual "0" "1" are all ways around the problem but don't really solve it and increase packet size. also why would we assume that characters would be encoded differently at the other end all CC computer are the same and should interpret charterer sets in the same way. some file types like MIDI and png / bmp could be used on CC computers encryptions is also a reason and ziping a file is pointless if it is going to make the packet bigger when base64 converts it.

If it is too much trouble to fix then don't worry about it but if there is a simple solution it would be nice to have.
Quote

Cloudy's Photo Cloudy 25 Oct 2012

A simple solution is base64 ^_^/>

Packet size is not important in a system where packet size is unlimited.
Quote
Locked