Jump to content




CC Tar - create and unpack (real) tar files in cc

lua utility

23 replies to this topic

#1 columna1

  • Members
  • 65 posts

Posted 23 April 2014 - 12:59 PM

- CC Tar -

by Columna1


I have spent a lot of time making this and had a hard time deciding to release it or not but here it is my tar program.

I have built in an automatic updater so if some bug comes up I should be able to fix it and everyone will get the new version automatically.

As of now It requires http to pack up files but I can change that to just give a warning if you guys want. changed - no longer requires http to run

*WARNING* This program makes and reads BINARY FILES so you CAN'T get them through the http api with normal methods (like http.get("http://www.mysite.com/test.tar") because of the way luaj handles strings!!! so you have to make a custom downloader (in a completely new way) or drag and drop the files in the computer's save folder. I'm not sure what will happen if you read a file download that way so don't blame me if something happens
-if you guys want I can also explain how I get around this limitation but it isn't easy-


To use you launch the program as so
tar c [directory] [output file] or tar u [input file]

aka

tar c dir dirtest.tar
or
tar u dirtest.tar

This program uses absolute paths so just remember that when you try to use it.

These tar files are REAL so they can be opened with programs like 7zip, winrar, GNU tar, and others etc...
this format is not compressed at all so sometimes It has problems with large files but Ive tried to fix them, If you have and legit solutions (aka. look at the code first before saying anything) you can tell me and I can fix it.

here is the pastebin
pastebin.com/bcgmNT2Q
or
pastebin get bcgmNT2Q tar

(I can post screenshots but there isn't much to show if anything so if you wana see it just download it)

Have fun using this feel free to use in one of your own projects but please don't claim as your own!
(I have many other cool programs like a proper midi reader and an nbs reader/player and Im afraid to release them because some were weeks of work so if you guys can prove me wrong and show you that you wont rip my programs off I might release them too)

Edited by columna1, 29 June 2018 - 08:50 PM.


#2 wilcomega

  • Members
  • 466 posts
  • LocationHolland

Posted 23 April 2014 - 05:16 PM

neat, i liek seeing real life file types getting a bit of support in CC

#3 awsmazinggenius

  • Members
  • 930 posts
  • LocationCanada

Posted 24 April 2014 - 12:06 AM

Nice work! Don't be afraid to release your code, nobody should just copy-paste it and call it theirs (I think it's only happened once or twice with CC code.)

I've wanted to do something like this in CC for a while with zip files, but too many projects and I wouldn't want to use the HTTP API, so either I'd have to implement zip compression/decompression in Lua or find another implementation that isn't just a wrapper over C code.

#4 columna1

  • Members
  • 65 posts

Posted 24 April 2014 - 12:37 AM

View Postawsmazinggenius, on 24 April 2014 - 12:06 AM, said:

I wouldn't want to use the HTTP API, ...

Umm are you implying Im packing things elsewhere, I immagine not but if so then no Im not packing the files elsewhere I use the http to get the timestamp for the file (so that it is accurate) but It isnt needed per say you can use an old timestamp.

Edited by columna1, 24 April 2014 - 12:38 AM.


#5 Agoldfish

  • Members
  • 451 posts
  • LocationSome Fish Bowl in Ohio.

Posted 24 April 2014 - 12:39 AM

This is interesting. Too bad I never use .tar format. :P

#6 theoriginalbit

    Semi-Professional ComputerCrafter

  • Moderators
  • 7,332 posts
  • LocationAustralia

Posted 24 April 2014 - 12:53 AM

Nice. I'd like to see some compression going on, not just packing, but its good to see some more real standards being implemented.

As for downloading and uploading the tar files, its not that hard at all really, to base64 the package on the sending side, send it, then from base64 on the receiving side and you're done... NeverCast and I used this method with one of our projects when we stumbled across the LuaJ bug.

#7 Sir_Mr_Bman

  • Members
  • 62 posts

Posted 24 April 2014 - 09:48 PM

I like the idea.

I don't like .tar

Still, nice work on this!

#8 columna1

  • Members
  • 65 posts

Posted 25 April 2014 - 08:09 PM

View Posttheoriginalbit, on 24 April 2014 - 12:53 AM, said:

Nice. I'd like to see some compression going on, not just packing, but its good to see some more real standards being implemented.

As for downloading and uploading the tar files, its not that hard at all really, to base64 the package on the sending side, send it, then from base64 on the receiving side and you're done... NeverCast and I used this method with one of our projects when we stumbled across the LuaJ bug.

I'm not so sure about real compression as proper compression is hard with the languages that have proper binary support let alone lua...

View PostAgoldfish, on 24 April 2014 - 12:39 AM, said:

This is interesting. Too bad I never use .tar format. :P

View PostSir_Mr_Bman, on 24 April 2014 - 09:48 PM, said:

I like the idea.

I don't like .tar

Still, nice work on this!

Do you realise that you can use this with programs like winrar, 7zip and so on? many of the popular (and even unpopular) archive readers can and will open/make tar files. Its a nice way to transfer a lot of files from one place to another (think about computercraft without wildcards, you have a computer with a lot of files in it, all you would do is tar c / comp.tar and then you can move the comp.tar onto the floppy drive and to another computer and etc...)
or you have a program that uses a lot of directories you can make a package and such.
plus the biggest part I would say is that
you can make a small program like theorigionalbit said that converts things into base64 (or even make it convert straight into hex which is easier) and upload a whole file to pastebin to download it on another server and convert it back to use with tar
(heck if you wanted me to I could make that for you)

Edited by columna1, 26 April 2014 - 10:39 PM.


#9 Agoldfish

  • Members
  • 451 posts
  • LocationSome Fish Bowl in Ohio.

Posted 25 April 2014 - 08:12 PM

View Postcolumna1, on 25 April 2014 - 08:09 PM, said:

View Posttheoriginalbit, on 24 April 2014 - 12:53 AM, said:

-snip-
-snip-

View PostAgoldfish, on 24 April 2014 - 12:39 AM, said:

-snip-

View PostSir_Mr_Bman, on 24 April 2014 - 09:48 PM, said:

-snip-
-snip-
I guess I should have said: "I never download .tar files unless I have to."

#10 ardera

  • Members
  • 503 posts
  • LocationGermany

Posted 27 April 2014 - 09:06 AM

View Postawsmazinggenius, on 24 April 2014 - 12:06 AM, said:

Nice work! Don't be afraid to release your code, nobody should just copy-paste it and call it theirs (I think it's only happened once or twice with CC code.)

I've wanted to do something like this in CC for a while with zip files, but too many projects and I wouldn't want to use the HTTP API, so either I'd have to implement zip compression/decompression in Lua or find another implementation that isn't just a wrapper over C code.
I've tried this too actually, but the documentation of zip files on PKWARE is bad. I don't know if it's just me, but it's like every zip version has a different structure. The Deflation or Deflation64 (or some other compression algorithm) is not the most difficult on extracting zip's, it's a common used algorithm and you find many documentations about it, but zips are just way to downwards-incompatible, it would take ages to implement all structures.

#11 skwerlman

  • Members
  • 163 posts
  • LocationPennsylvania

Posted 17 May 2014 - 01:45 AM

Maybe if someone gets a working .gz (un)archiver, we can have a working apt-get system :D

#12 CodingWithClass

  • Members
  • 17 posts

Posted 24 May 2014 - 01:36 AM

Could I include this program in my up-and-coming UNIX-like OS?

#13 columna1

  • Members
  • 65 posts

Posted 26 May 2014 - 12:04 AM

View PostCodingWithClass, on 24 May 2014 - 01:36 AM, said:

Could I include this program in my up-and-coming UNIX-like OS?
Sure thing just make sure to give me credit

-also if you want any changes just ask me and I can do it

Edited by columna1, 26 May 2014 - 12:05 AM.


#14 Sir_Mr_Bman

  • Members
  • 62 posts

Posted 26 May 2014 - 01:52 AM

Oh hey!

I noticed a bug!

--get time
  if not http then
	error("Please enabe the http api")
  end
  --local time = "1397584826"
  print("grabbing time...")
  local res = http.get("http://www.timeapi.org/utc/now?format=%25s")

Regardless of if the HTTP API is off or not, the program will continue to try and get the timestamp.

Might want to throw in a local time stamp...
Something like this maybe(?):
local time
if not http then
  print("HTTP API is disabled. Please enable that.")
  print("For now, we'll just ignore this and throw in a local time.")
  time = 1397584826
else
  print("Fetching time!")
  local web = http.get("http://www.timeapi.org/utc/now?format=%25s")
  time = tonumber(web.readAll())
  -- The rest of the code...
end

Something like that maybe. I dunno.

Edit: Un-derped code.

Edited by Sir_Mr_Bman, 26 May 2014 - 01:55 AM.


#15 CodingWithClass

  • Members
  • 17 posts

Posted 26 May 2014 - 12:22 PM

View Postcolumna1, on 26 May 2014 - 12:04 AM, said:

View PostCodingWithClass, on 24 May 2014 - 01:36 AM, said:

Could I include this program in my up-and-coming UNIX-like OS?
Sure thing just make sure to give me credit

-also if you want any changes just ask me and I can do it
Perfect, will do!

#16 columna1

  • Members
  • 65 posts

Posted 27 May 2014 - 11:06 AM

View PostSir_Mr_Bman, on 26 May 2014 - 01:52 AM, said:

Oh hey!

I noticed a bug!

--get time
  if not http then
	error("Please enabe the http api")
  end
  --local time = "1397584826"
  print("grabbing time...")
  local res = http.get("http://www.timeapi.org/utc/now?format=%25s")

Regardless of if the HTTP API is off or not, the program will continue to try and get the timestamp.

Might want to throw in a local time stamp...
Something like this maybe(?):
local time
if not http then
  print("HTTP API is disabled. Please enable that.")
  print("For now, we'll just ignore this and throw in a local time.")
  time = 1397584826
else
  print("Fetching time!")
  local web = http.get("http://www.timeapi.org/utc/now?format=%25s")
  time = tonumber(web.readAll())
  -- The rest of the code...
end

Something like that maybe. I dunno.

Edit: Un-derped code.
error() makes the code stop in its tracks so it wont continue, and I didn't like the idea of having an outdated timestamp but ill update it because someone asked

Edited by columna1, 27 May 2014 - 10:03 PM.


#17 RoD

  • Members
  • 313 posts

Posted 13 June 2014 - 11:52 PM

How did you know how tar compression works? Can you explain a little of the code and where did you get that information? I am really amazed. Just..

#18 columna1

  • Members
  • 65 posts

Posted 14 June 2014 - 01:28 AM

View PostRoD, on 13 June 2014 - 11:52 PM, said:

How did you know how tar compression works? Can you explain a little of the code and where did you get that information? I am really amazed. Just..
Sure, http://www.gnu.org/s...e/Standard.html that is where I got most of the information and what about it would you like me to explain?
btw tar files are an un-compressed archive format meaning that there was no complicated compression, just file headers, and then the data after that

EDIT: oh, also I used a program called HxD to look at a hex representation of some existing tar files to help me find some stuff out

Edited by columna1, 14 June 2014 - 02:17 AM.


#19 RoD

  • Members
  • 313 posts

Posted 14 June 2014 - 10:33 AM

View Postcolumna1, on 14 June 2014 - 01:28 AM, said:

View PostRoD, on 13 June 2014 - 11:52 PM, said:

How did you know how tar compression works? Can you explain a little of the code and where did you get that information? I am really amazed. Just..
Sure, http://www.gnu.org/s...e/Standard.html that is where I got most of the information and what about it would you like me to explain?
btw tar files are an un-compressed archive format meaning that there was no complicated compression, just file headers, and then the data after that

EDIT: oh, also I used a program called HxD to look at a hex representation of some existing tar files to help me find some stuff out
I see, yesterday i used my hxaeditor and saw that there is no compression, just hexadecimal code. Still, pretty awesome :D

#20 columna1

  • Members
  • 65 posts

Posted 14 June 2014 - 02:51 PM

View PostRoD, on 14 June 2014 - 10:33 AM, said:

View Postcolumna1, on 14 June 2014 - 01:28 AM, said:

*snip*
I see, yesterday i used my hxaeditor and saw that there is no compression, just hexadecimal code. Still, pretty awesome :D
welll I think you misunderstand. even if there was compression there would still be hexadecimal code as the hex editor just shows you a hexadecimal representation of the binary data. this allows people who know what they are looking for to find out what each byte is and see how something is set up. Kinda like looking at a .txt save file with just a bunch of numbers except this stuff is in binary and cant be read like text.





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users