nil is Lua's null. It is different from any other value and represents false in a logical expression. Most of the time, it is used to indicate the lack of a useful value.
Check Wikipedia's page on the null pointer for more informations.
For a more in-depth and concise explanation of types, please consult the Lua 5.1 Official Manual.