Jump to content




Number conversion


2 replies to this topic

#1 TheZen

  • Members
  • 65 posts
  • LocationHoquiam Wa

Posted 28 June 2019 - 07:32 AM

Basically I want to make an accurate way of converting numbers between bases
I like to test my method using base 10 to 16 and 10 to 16
but when the numbers get a little larger (34A BFC6 3278 ACBF) specificaly in this case there is a lack of accuracy...
if I convert 34ABFC63278ACBF to decimal and back i get 34ABFC63278ACC0 which is 1 greater than 34ABFC63278ACBF is.

before you tell me to use string.format() or tonumber(), like I have seen others do on the internet, they also have the same problem.

is there any way y'all could think if to make it more accurate?

PS. also not asking for, "No one needs to be that accurate"

Edited by KidBrine, 28 June 2019 - 07:34 AM.


#2 SquidDev

    Frickin' laser beams | Resident Necromancer

  • Members
  • 1,427 posts
  • LocationDoes anyone put something serious here?

Posted 28 June 2019 - 08:11 AM

Lua, and so ComputerCraft represents numbers using doubles. As a result, when you start working with large numbers, you'll begin to lose some precision. 0x34ABFC63278ACBF is in the range of 257, and so is well beyond the region that doubles can represent accurately (doubles can handle integers up to 252 for those curious). There's not really a nice solution to this, so your best bet is to use one of the many big integer libraries out there instead. They effectively split the number up into smaller bits and store it in a table, allowing you to have arbitrarily large numbers.

#3 TheZen

  • Members
  • 65 posts
  • LocationHoquiam Wa

Posted 28 June 2019 - 05:54 PM

 SquidDev, on 28 June 2019 - 08:11 AM, said:

Lua, and so ComputerCraft represents numbers using doubles. As a result, when you start working with large numbers, you'll begin to lose some precision. 0x34ABFC63278ACBF is in the range of 257, and so is well beyond the region that doubles can represent accurately (doubles can handle integers up to 252 for those curious). There's not really a nice solution to this, so your best bet is to use one of the many big integer libraries out there instead. They effectively split the number up into smaller bits and store it in a table, allowing you to have arbitrarily large numbers.

And I now need to make them CC friendly.





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users