Introduction

This is the documentation for using the lua-sys library for LuaJIT.

Installation

First, set up lde.

Note

If you are unsure, please consult lde’s documentation on adding dependencies and usage of lde.

lde add lua-sys

Example

Here’s a basic example to run some lua code.

local lua = require("lua-sys")

local state = lua.new()
print(state:eval("return 1 + 2")) -- 3

Next Steps

Look on the sidebar for guides for using different parts of lua-sys!

Passing A Function to Lua

You can give host functions to the guest state. Guest code can call them.

local lua = require("lua-sys")

local state = lua.new()

state:globals().adder = function(a, b)
	print("Called by guest!")
	return a + b
end

print(state:eval("return adder(5, 5)")) -- 10

The bridge converts the arguments for the basic types: numbers, strings and booleans. Refer to How values cross the boundary.

A guest function that the host receives becomes a normal host function. This event occurs when a guest function is a result of a call, or a value in a guest table:

local double = state:eval("return function(x) return x * 2 end")
print(double(21))  -- 42

A guest function as an argument of a host callback is different. An argument of the type function, userdata or thread gives this error:

bridge: cannot pass function across independent states

Put the value in a guest table and pass the table instead.

Tables

A table is different from a function. The bridge does not copy a table into a host value, for three reasons:

1. A table can change at any time. Fields can come and go.
2. Because of reason 1, a copy is not correct.
3. A copy is slow.

Therefore, the host keeps a table as a reference and reads fields as necessary.

The call state:globals() shows this design. It returns a lua.Table for the guest global table. The assignment state:globals().adder = ... above uses that proxy.

To identify a lua.Table, use type(x) == "table". A guest table always arrives as a lua.Table, never as a plain host table.

Tip

For more information, refer to the API Reference.

Sandboxing

This page states what a guest state isolates, what it does not isolate, and what a host program must add before it can run code that it does not trust.

What the guest cannot do

• The guest cannot see host globals, host modules or host functions. The bridge copies primitives and passes compound values by reference, so a value reaches the other side only through the API.
• The guest cannot corrupt the host state. An error in the guest returns as a string, and the host stack stays balanced.
• A limit on memory is exact and is enforced by the allocator of the state. Refer to state:setMemoryLimit().
• A count hook gives a reliable instruction limit while the debug and jit libraries are out of the guest. Refer to the warning below.

What a fresh guest state does not isolate

lua.new() opens every standard library, which is correct for a cooperating guest and wrong for an untrusted one. These facts are measured on the current version:

A watchdog is therefore only reliable when the guest has no jit library and no debug library. Both of them give the guest control over the hook mechanism itself.

The recipe for an untrusted guest

Run the code in a state that you prepare, and remove every path out:

1. Build the global table for the guest by hand. Keep the safe core: assert, error, pcall, xpcall, select, type, tostring, tonumber, next, pairs, ipairs, unpack, setmetatable, getmetatable, rawget, rawset, rawequal, coroutine, table, string, math and bit.
2. Leave out io, os, package, require, debug, jit, ffi, loadfile, dofile, getfenv, setfenv and string.dump.
3. Provide load and loadstring as host functions that use chunk:setMode("text"), so the guest cannot load bytecode.
4. Set a memory limit with state:setMemoryLimit().
5. Set a deadline with state:setHook() and a count mask, with no jit and no debug in the guest.
6. Know the depth of the call chain. The bridge limits it to 200 levels, and state:setChainLimit() changes that figure. A guest that recurses inside a tail call keeps its own stack flat, so without this limit it can stop the process through the C stack.
7. Cap the output that host callbacks give back to the guest, for example the bytes of a print function, because the guest controls how often it calls them.

The boundary of this library

lua-sys isolates two lua_State instances from each other. It is not a security boundary for the LuaJIT virtual machine itself:

• A hostile bytecode chunk can still damage the interpreter, because LuaJIT trusts its own bytecode. Refuse bytecode with a text mode.
• A long chain of host to guest to host calls uses C stack space for each step. The bridge limits the chain to 200 levels by default, which is safe for a thread stack of 1 MB. Refer to state:setChainLimit().
• The library gives no limit on execution time by itself. Use a hook, as in step 5 above.

For code that must not damage the host under any condition, add the operating system as a second boundary: run the whole program, or a child process, with a memory cap and a time limit from the outside. The in-process limits above then stop most accidents before they reach the operating system.

API Reference

lua-sys gives you the Lua C API of LuaJIT. Use it to make guest lua_State instances and to control them from the host program.

local lua = require("lua-sys")

The module has three members:

The high-level API below is the approved method to control a guest state. Values cross between the host and the guest in two ways: by copy for primitives, or by reference for tables, functions, userdata and threads.

Quick index

How values cross the boundary

The guest is an independent interpreter. It has its own heap, its own globals, its own package.loaded and its own JIT engine. Nothing is shared with the host. Each value that crosses is a copy or a reference.

Host → guest

Guest → host

Thus type(x) on the host is a reliable test. A guest table always arrives as a host table, and this value is always a lua.Table.

Arguments of host callbacks

Guest code can call a host function. The bridge converts the arguments from guest values to host values. Only primitives and tables can cross in this direction. A function, a userdata or a thread causes this error:

bridge: cannot pass <type> across independent states

To send these values to the host, put them in a guest table and pass the table.

Return values of host callbacks

A host callback can return primitives only: nil, boolean, number and string. The bridge copies each returned value into the guest. This error occurs if the callback returns a table, a lua.Table, a function or another compound value:

bridge: host callback returned a <type>; only primitives (nil, boolean, number, string) can be returned from host to guest

To give structured data to the guest, write the data into the guest. You can do one of these two steps:

• Set a guest global.
• Let the guest pass a table to the callback. The callback gets a live lua.Table view and can change the table.

state:globals().fill = function(t)
    t.timeout = 5          -- t is the guest table from the guest
    t.retries = 3
    return true            -- a primitive result is correct
end

state:eval("local cfg = {}; fill(cfg); return cfg.timeout")  -- 5

lua.new() → lua.State

Makes a new guest lua_State. These libraries are open in the new state: the base library, package, coroutine, table, string, math, io, os, debug, bit and jit.

The modules ffi and string.buffer are not preloaded. However, require("ffi") and require("string.buffer") operate in the guest.

The new state is fully independent of the host interpreter.

local state = lua.new()
print(state:eval("return _VERSION"))       -- Lua 5.1
print(state:eval("return jit.version"))    -- LuaJIT 2.1.x
state:close()

Close the state with state:close() when you do not need it again.

state.L → lua_State*

The raw guest state pointer, as an FFI cdata value. Use it with lua-sys.raw, with ffi.C calls, or to compare it with info.thread in a hook. The value is nil after the state is closed.

state:close()

Closes the guest state. The operation releases the heap, the registry references and all host callbacks of the state. A second call is safe.

All lua.Table objects, lua.Value objects and guest callables of the state become invalid. Their registry references stop with the state.

Note

Each operation on a closed state gives the error state is closed. This rule covers state:eval, state:load, state:globals, state:table, the lua.Chunk methods, the lua.Table methods and calls to a guest callable. A call with fn:pcall() gives false, "state is closed". A stored frame gives neutral results, as it does after the hook returns. The process does not stop.

Close the state one time only, at the end of its life. Do not use the objects of a closed state.

state:setChainLimit(levels)

Limits the depth of a nested host to guest chain. The default is 200 levels, and the value 0 removes the limit.

Each level of a chain, for example a host callback that calls a guest function that calls a host callback, uses C stack. A chain that is too deep stops the process, and a guest can build such a chain on purpose: write the guest handler with a tail call, and the guest Lua stack stays flat while the C stack grows.

state:setChainLimit(500)

local ok, err = pcall(handler, 5000)
-- ok == false, err contains "cross-state call chain reached its limit"

The error follows the usual rules. A guest function raises it on the host, and fn:pcall() returns it. Inside the guest, a pcall around the call that reaches the limit catches it. The depth is restored on each exit, so the state keeps its full budget after the error.

state:chainDepth() → integer

The number of levels of the host to guest chain that are active now. The value is 0 outside a chain, and 1 inside a single host callback called from the guest. Use it for diagnostics.

state:memory() → integer

The number of bytes that the guest state holds now. The number comes from the allocator of the state, so it is exact. A fresh state starts at about 35 KB, because the standard libraries are open.

local state = lua.new()
print(state:memory())            -- about 35960

local big = state:eval([[return string.rep("x", 1000000)]])
print(state:memory())            -- about 1 MB more

big = nil
state:eval("collectgarbage('collect')")
print(state:memory())            -- lower again

The count follows the blocks that the state holds, not the live data. After a collection, the state can keep memory for later use, so the value does not always fall to the earlier figure.

state:setMemoryLimit(bytes)

Limits the guest state to bytes. An allocation that would pass the limit fails, and the guest gets the error not enough memory. The value 0 removes the limit, which is the default. The two limits of a state are separate, and a limit does not stop state:close().

state:setMemoryLimit(state:memory() + 200000)

local ok, err = state:load([[return string.rep("y", 2000000)]]):pcall()
-- ok == false, err contains "not enough memory"

state:setMemoryLimit(0)          -- no limit again

The guest can catch the error and continue with less memory, so treat the limit as a control on the size of the state, and combine it with a hook for the execution time.

Evaluating code

state:load(code [, chunkName]) → lua.Chunk

Wraps Lua source code in a builder. You can set up the builder before the code runs. The operation does not compile the code. Therefore, a syntax error occurs at the call that runs the chunk, not at state:load().

chunkName sets the chunk name for the debug information. Guest code sees this name as debug.getinfo(1, "S").source. Use the prefix @ for a file path, for example "@/path/to/file.lua".

The argument code can be source text or LuaJIT bytecode. A chunk accepts both by default. Refer to chunk:setMode() to refuse one format, and to chunk:isBytecode() to examine the source.

state:eval(code [, chunkName]) → value

Compiles and runs code immediately. The method returns the first result. It returns nil if the chunk has no result. This method is the same as state:load(code, chunkName):eval().

The method first compiles the code as return <code>. If the result is a syntax error, it compiles the original source. Thus a bare expression operates, and a block of statements operates also:

local state = lua.new()

print(state:eval("1 + 2"))                        -- 3
print(state:eval("return 1 + 2"))                 -- 3
print(state:eval("local x = 5; return x * 2"))    -- 10
print(state:eval("local x = 5"))                  -- nil

An error in the guest, and a syntax error, becomes a string error on the host. Use pcall to catch it:

local ok, err = pcall(function() return state:eval("error('boom')") end)
-- ok == false, err == "boom"

lua.Chunk

The builder from state:load(). It holds the source code, an optional chunk name, an optional mode and the owner state.

The chunk compiles one time, at its first run, and keeps the result. Later runs use the compiled function again, so a run costs about the same as a call to a guest function. A chunk that returns a function still creates a new closure at each run. A change of the name or of the mode compiles the chunk again.

A chunk accepts source text and LuaJIT bytecode by default. Use chunk:setMode() to refuse one format, and chunk:isBytecode() to examine the source before the chunk runs.

chunk:eval(...) → value

Compiles and runs the chunk. The arguments become ... in the guest. The method returns the first result, or nil if the chunk has no result. An error in the guest becomes an error on the host.

local chunk = state:load("return ...")
print(chunk:eval("hello"))   -- hello
print(chunk:eval("world"))   -- world

chunk:call(...)

Compiles and runs the chunk. The arguments become ... in the guest. The method discards all results. Use it for a script with side effects.

state:load("print('hello from guest')"):call()
state:load("_sum = select('#', ...)"):call(1, 2, 3)
print(state:globals()._sum)  -- 3

chunk:pcall(...) → true, ... | false, err

The same as :eval(), but the method returns errors and does not raise them. The result is true and all results after a success, or false, err after a failure. The method also catches a syntax error, because the compilation occurs inside the protected call.

local ok, a, b = state:load("return ... + 1, ... * 2"):pcall(10)
-- ok == true, a == 11, b == 20

local ok2 = state:load("local x = 1"):pcall()
-- ok2 == true, because a chunk without a result gives only true

local ok3, err3 = state:load("1 +"):pcall()
-- ok3 == false, err3 == "[string \"1 +\"]:1: unexpected symbol near '1'"

local ok4, err4 = state:load("error('boom')"):pcall()
-- ok4 == false, err4 == "boom"

chunk:xpcall(...) → true, ... | false, err

This method has the same result as :pcall(). In addition, the error string contains a guest stack traceback after a failure. The guest debug.traceback is the error handler for the call, so the traceback is complete before the stack clears.

local ok, err = state:load("error('boom')"):xpcall()
-- ok == false
-- err == "boom\nstack traceback:\n\t..."

Use :xpcall() to report an error in guest code with context. Use :pcall() when the message alone is sufficient.

chunk:setName(name) → lua.Chunk

Sets the chunk name for the debug information. The method returns the chunk, so you can add more calls. A new name makes the next run compile again. This method is the same as the argument chunkName of state:load().

local src = state:load("return debug.getinfo(1, 'S').source")
    :setName("@myscript.lua")
    :eval()
print(src)  -- @myscript.lua

chunk:setMode(mode) → lua.Chunk

Sets the format that the chunk accepts. The method returns the chunk, so you can add more calls. A new mode makes the next run compile again.

The Lua mode letters "t", "b" and "bt" are also correct, and "binary" is an alias of "bytecode". The value is not case sensitive.

A mismatch shows at the call that runs the chunk. The error is attempt to load chunk with wrong mode:

local bytecode = state:eval("return string.dump(function() return 7 end)")

state:load(bytecode):eval()                     -- 7, the mode is "both"
state:load(bytecode):setMode("text"):eval()     -- raises "wrong mode"
state:load(bytecode):setMode("bytecode"):eval() -- 7
state:load("return 1"):setMode("bytecode"):eval() -- raises "wrong mode"

The method chunk:pcall() returns the error instead of raising it:

local ok, err = state:load(bytecode):setMode("text"):pcall()
-- ok == false, err == "attempt to load chunk with wrong mode"

An unknown mode gives setMode: unknown mode "<mode>" (expected "text", "bytecode" or "both"). A value that is not a string gives setMode: mode must be "text", "bytecode" or "both", got <type>.

chunk:getMode() → "text" | "bytecode" | "both"

The format that the chunk accepts. The value is "both" until you call setMode().

print(state:load("return 1"):getMode())                    -- both
print(state:load("return 1"):setMode("text"):getMode())    -- text

chunk:isBytecode() → boolean

true when the source of the chunk starts with the escape byte that marks a precompiled chunk. Use this method to examine data from a source that you do not trust, before the chunk runs.

LuaJIT loads bytecode with the signature \27LJ. Other binary data also starts with the escape byte. The loader refuses such data with cannot load incompatible bytecode or cannot load malformed bytecode.

local chunk = state:load(source)

if chunk:isBytecode() then
    error("bytecode is not permitted")
end

chunk:setMode("text"):call()

The mode does not change the result of isBytecode().

chunk(...)

The __call metamethod. A direct call to a chunk is the same as chunk:eval(...).

print(state:load("return ... * 2")(21))  -- 42

Guest function callables

A guest function arrives on the host as an ordinary host function. This event occurs when the function is a result of state:eval(), :load():eval(), chunk:pcall(), Table:get() or an iteration with pairs(). The host function calls back into the guest:

local add = state:eval("return function(a, b) return a + b end")
print(add(1, 2))  -- 3

local double = state:eval("return function(x) return x * 2 end")
state:globals().double = double          -- and back into the guest
print(state:eval("return double(21)"))   -- 42

Arguments and results obey the rules above. The bridge copies primitives, and it passes tables by reference. An error in the guest raises an error on the host.

A guest function has one callable for the life of its state. Two fetches of the same guest function therefore give the same host function, and == is true. Two proxies of one guest table are different objects. Compare values in the guest if identity is important.

fn:pcall(...) → true, ... | false, err

Calls the guest function with protection. The result is true and all results, or false, err when the guest function raises an error. The host gets no error.

local fn = state:eval("return function(x) return x * 2, x + 1 end")
local ok, a, b = fn:pcall(21)
-- ok == true, a == 42, b == 22

local boom = state:eval("return function() error('kaboom') end")
local ok2, err = boom:pcall()
-- ok2 == false, err == "[string \"return function() error('kaboom') end\"]:1: kaboom"

A guest function without a result gives true only. The method comes from a function metatable, so the field fn.pcall is also available.

Globals and tables

state:globals() → lua.Table

Returns a lua.Table proxy for the global environment of the guest (_G):

local g = state:globals()
g.myVar = 42
print(state:eval("return myVar"))  -- 42
print(g.print == nil)              -- false, because print is a callable

Each call gives a new proxy for the same guest table. Two proxies are never ==. Keep one proxy, or compare the guest tables in the guest.

state:table([init]) → lua.Table

Makes a new empty guest table. The argument init is optional and is a plain host table. The new table is not registered. Assign it to a global, return it, or pass it to guest code.

local t = state:table({
    name = "alice",
    pos  = { x = 1, y = 2 },
    greet = function(n) return "hi " .. n end,
})

print(t.name)                     -- alice
print(t.pos.x)                    -- 1
print(t:get("greet")("world"))    -- hi world

The bridge converts the keys and the values of init with the normal host → guest rules:

state:table() gives these errors:

The cycle test tells a back edge from a duplicate. One table as two sibling values gives two copies and no error. A table that contains itself gives an error. A circular structure is not possible across state boundaries, so the bridge rejects it.

Table:get(key) → value

Reads a key. Primitive values come back directly. A guest function comes back as a callable, a nested table as a lua.Table proxy, and a userdata or a thread as a lua.Value. A key that is absent gives nil.

The key can be a primitive or a guest value, such as a lua.Table, a callable or a lua.Value.

Table:set(key, value)

Writes a key. The value can be a primitive, a host function, a guest callable, a lua.Table or lua.Value reference, or a plain host table. A plain host table becomes a new guest table. A key with the value nil is removed.

Table field access

A lua.Table proxy sends field reads to :get() and field writes to :set(). Thus tbl.key is the same as tbl:get("key"), and tbl.key = v is the same as tbl:set("key", v):

local g = state:globals()
g.myVar = 42                     -- g:set("myVar", 42)
g.config = { timeout = 5 }       -- plain table becomes a guest table
print(g.myVar)                   -- 42
print(g.config.timeout)          -- 5

A method name has priority over a guest key. The names get, set, pairs, ipairs, type, value and free always give the method of the proxy. To read a guest key with one of these names, use :get() with a string, for example t:get("type").

Table:pairs() → iterator

Gives a stateless iterator for all key/value pairs of the guest table. It is the same as pairs() on a plain table, and it uses the next function of the guest:

for k, v in t:pairs() do
    print(k, v)
end

Table:ipairs() → iterator

Gives a stateless iterator for the integer keys 1..n. The iteration stops at the first nil. It is the same as ipairs():

for i, v in t:ipairs() do
    print(i, v)
end

Warning

Use the methods of the proxy. Do not use the host operators #t, pairs(t) and ipairs(t). These operators act on the host wrapper table of the proxy and its internal fields _state, _ref and _type. The result is incorrect for the guest table. For example, #t gives 0, and pairs(t) iterates the internal fields. The function next(t) is also incorrect. Always use t:pairs() and t:ipairs(). Use the # operator in the guest when you need a length.

lua.Value

A reference to a guest value without its own host proxy: userdata, lightuserdata and thread. lua.Table has the same methods.

local out = state:eval("return io.stdout")
print(type(out), out:type())   -- table, userdata
print(tostring(out))           -- lua.userdata
out:free()                     -- release the registry reference now

Value:type() → string

The name of the guest type: "nil", "boolean", "number", "string", "table", "function", "userdata", "lightuserdata" or "thread". For a lua.Table, the result is "table".

Value:value() → any

The value behind the reference. For a live reference, the method returns the object itself. The guest value is not copied. After free() or state:close(), the method returns the stored plain value. That value is nil for a referenced type.

Value:free()

Releases the guest registry reference immediately. The garbage collector also releases the reference at collection of the proxy (__gc). Use free() to reclaim memory early. A program that uses many guest values for a long time benefits from this call.

free() is safe to call more than one time. It is also safe after state:close().

Debug hooks and frames

state:setHook(fn, mask [, count])

Installs a debug hook on the guest state. This method is the high-level equivalent of the raw lua_sethook. It needs no FFI cast and no raw callback.

The bridge calls fn as fn(event, info) at each event:

• event is "call", "return", "line", "count" or "tailcall".
• info is a lua.HookInfo table for the event.

mask selects the events. Use a string with event names and spaces, or an integer bitmask:

state:setHook(function(event, info) end, "line")
state:setHook(function(event, info) end, "call return")
state:setHook(function(event, info) end, 4)          -- the same as "line"
state:setHook(function(event, info) end, "count", 1000)

count is the instruction interval for the "count" event. The default value is 1. A new hook replaces the old hook. The call state:setHook(nil) removes the hook.

An incorrect argument gives one of these errors:

The hook is on the guest state, not on the host. It operates while guest code runs, including code in a guest coroutine.

Note

LuaJIT fires hooks on interpreted code only. Thus the guest JIT engine is off while a hook is installed, and the bridge flushes the traces. Removal of the hook starts the engine again. A hook decreases the speed of hot guest code.

A hook that calls error() stops the running guest code with that error. A pcall around the call that started the guest code catches the error, and chunk:pcall also catches it. Therefore, a count hook is a simple timeout check:

state:setHook(function()
    error("timeout: guest code ran too long")
end, "count", 1000)

local ok, err = pcall(function()
    state:eval("while true do end")
end)
-- ok == false, err contains "timeout"

lua.HookInfo

The info table of the hook callback. The bridge fills all debug fields at the event, so the fields stay readable after the callback returns.

The metatable of the table supplies the method info.stack. It is not a stored field. Refer to info:stack().

info.thread is a lightuserdata with the thread pointer. Cast it to examine that thread with the raw API. This method is necessary for the frames of a coroutine, because lua_getstack requires the running thread:

local ffi = require("ffi")

state:setHook(function(event, info)
    if ffi.cast("lua_State*", info.thread) ~= state.L then
        -- the hook fired in a coroutine
    end
end, "line")

info:stack() → lua.Frame[]

Returns the stack trace of the thread as an array of lua.Frame objects. Item 1 is level 0, which is the frame of the hook event. The last item is the outermost frame.

state:setHook(function(event, info)
    for i, frame in ipairs(info:stack()) do
        print(i, frame.what, frame.source, frame.currentline)
    end
end, "line")

Warning

info:stack() examines the thread while it is paused at the hook. Call it inside the callback only. A later call raises info:stack() must be called from within the hook callback. The debug fields stay readable after the callback. The frames become inactive as described in lua.Frame.

lua.Frame

One stack frame. A frame is valid while the thread is paused at the hook. A frame has the same debug fields as info, but no event field. It has these two more fields:

A frame also has methods to read and write its state. A local and an upvalue are copies that the bridge converts with the normal guest → host rules. Thus a table local arrives as a lua.Table view of the live guest table.

Frame:locals() → { name, value }[]

Lists the active locals of the frame, in order. The result is an empty table if the frame does not exist, for example after the hook returns.

Frame:getLocal(name) → value

Reads one local by name. The result is nil if the frame has no local with that name.

Frame:setLocal(name, value) → boolean

Writes to an active local. The result is true on a success, and false if the local does not exist or the frame is gone. You can write only to locals that are active at the hook point. The running program sees the change:

state:setHook(function(event, info)
    local frame = info:stack()[1]
    if frame:getLocal("marker") then
        frame:setLocal("marker", 777)   -- the guest reads 777 afterwards
    end
end, "line")

Frame:upvalues() → { name, value }[]

Lists the upvalues of the running function as { name, value } pairs.

Frame:getUpvalue(name) → value

Reads one upvalue of the running function. The result is nil if the function has no upvalue with that name.

Frame:setUpvalue(name, value) → boolean

Writes to an upvalue of the running function. The result shows if the bridge found the upvalue. A write to an upvalue changes the shared cell, so other closures over that cell see the new value.

Frame:eval(code) → true, any | false, err

Evaluates code with the locals and the upvalues of the frame in scope. The code runs in a new environment that contains the active locals and the upvalues. A read of another name continues to the environment of the running function. A write to a new name goes there also. After a success, the bridge writes assignments to existing locals and upvalues back into the frame. Thus frame:eval("x = 42") changes the running program:

state:setHook(function(event, info)
    local frame = info:stack()[1]
    if frame:getLocal("marker") then
        local ok, value = frame:eval("marker * 2")   -- reads the local
        frame:eval("marker = marker + 1")            -- writes it back
    end
end, "line")

The result is true, firstResult, or false, err after an error in the guest or a syntax error. Like the other frame methods, call it inside the hook callback only.

Frames after the hook returns

A frame is valid only while the thread is paused at the hook. After the hook returns, and after the close of its state, the methods give neutral results and do not stop the process:

JIT control

state:jitOff([fn]) → state, state:jitOn([fn]) → state, state:jitFlush()

These three methods control the JIT compiler of the guest. Without an argument, the method switches the complete engine of the state. With a guest callable of the same state, the method changes that function only. jitOff and jitOn return the state, so you can add more calls. jitFlush returns nothing.

state:jitOff()        -- disable the JIT for the complete guest state
state:jitOn()         -- enable it again
state:jitOff(fn)      -- disable compilation of one guest function
state:jitOn(fn)       -- enable that function again
state:jitFlush()      -- discard all compiled traces

An argument that is not a guest callable of this state gives jitOff: fn must be a guest callable obtained from this state. The method jitOn gives the same error. After close(), each of the three methods gives state is closed.

state:setHook disables the engine while a hook is installed and flushes the traces at removal. These methods give direct control. For example, you can keep the other functions of the state compiled while one function stays interpreted.

Profiler

local profiler = require("lua-sys.profiler")   -- also lua.profiler

A sampling profiler for guest states. It uses the profiler hooks of LuaJIT.

profiler.start(state [, mode] [, callback])

Starts the sampling of state, which must be an open lua.State. The method gives profiler.start: expected an open lua.State for nil, a plain table or an already closed state. It gives profiler already running for this state if the state is in the sampling mode already.

The profiler keys its bookkeeping by the state object and not by its address. A state that you close without profiler.stop therefore does not block the next state.

• mode is a LuaJIT profiler mode string. The default is "fi1". Use f for function-level stacks, l for line-level stacks, and i<ms> for the sampling interval in milliseconds.
• callback is optional and has the form function(stack, samples, vmstate). The bridge calls it at each sample. stack is the frame list with semicolons, samples is the number of samples in this interval, and vmstate is a one-character string for the VM state from LuaJIT, for example "N" or "I". A callback stops the aggregation of the samples, so stop returns nil.

profiler.stop(state) → report

Stops the sampling. The method returns the aggregated report, in order of sample count from high to low. The result is nil if the profiler started with a custom callback. A state without active sampling gives profiler not running for this state.

The report is an array of entries with an extra field total:

{
    { stack = "fib;fib;fib;main", vmstate = "N", count = 150, percent = 75.0 },
    { stack = "main",             vmstate = "I", count = 50,  percent = 25.0 },
    total = 200,
}

profiler.print(report [, out] [, min_percent])

Writes the report to out as a table. The default for out is io.stdout. The method hides entries below min_percent. The default is 1.

local state = lua.new()
local work = state:load("local function fib(n) if n < 2 then return n end return fib(n-1) + fib(n-2) end for i = 1, 300 do fib(20) end")

profiler.start(state)
work()
profiler.print(profiler.stop(state))
state:close()

lua-sys.raw

require("lua-sys.raw") is a thin FFI binding of the complete Lua 5.1 and LuaJIT C API. The module is also available as lua.raw. Each function takes a lua_State* as the first argument. Use state.L for a guest, or another state pointer. The high-level API uses this module.

local raw = require("lua-sys.raw")

local L = raw.lnewstate()
raw.openlibs(L)
raw.loadstring(L, "return 2 + 2")
raw.pcall(L, 0, 1, 0)          -- 0 == LUA_OK
print(raw.tonumber(L, -1))     -- 4
raw.close(L)

Naming

More examples of the raw.lX form: luaL_error → raw.lerror, luaL_checkstack → raw.lcheckstack, luaL_setmetatable → raw.lsetmetatable.

Differences from the C API

• The C API gives an integer for a status or a test. In raw, these results are Lua booleans: raw.equal, raw.rawequal, raw.lessthan, raw.next, raw.checkstack, raw.isnumber, raw.isstring, raw.iscfunction, raw.isuserdata, raw.isyieldable, raw.getmetatable, raw.setmetatable, raw.getfenv, raw.setfenv, raw.pushthread, raw.callmeta and raw.testudata.
• The module wraps the output parameters. raw.tolstring(L, idx) and raw.checklstring(L, idx) return a Lua string. raw.jit_profile_dumpstack(L, fmt, depth) copies the profiler buffer into a Lua string. The C buffer is valid until the next call only.
• The module has helpers for common sequences: raw.pop(L, n) is lua_settop(L, -n-1), and raw.getglobal(L, name) is lua_getfield(L, LUA_GLOBALSINDEX, name).
• state.L is a lua_State* cdata. Each raw function accepts it directly.

Caution

A raw call into a state uses the LuaJIT FFI. The FFI is not safe for re-entry across independent states. A guest state that you control with raw while guest code calls back into the host can stop the trace recorder of LuaJIT. Use the high-level API for all traffic between the host and a guest. The bridge sends each transition through a lua_CFunction, which the JIT treats as an opaque boundary. Refer to Bridge Design for the full explanation.

lua-sys.bridge

An internal compiled module (bridge.so, .dylib or .dll). Its functions support the high-level API: new_state, close_state, make_callable, push_callback, register, unregister, set_hook, remove_hook, compound_tag and set_frame_meta.

These functions are not a public interface. Their signatures and their operation can change without notice. Do not call them. Use lua-sys.raw when you need a function below the high-level API.

Rules and caveats

• Close the state one time, at the end. state:close() releases everything that belongs to the state, including the references in it. The guarded methods give state is closed. All other operations, such as the evaluation of code, table access and calls to guest functions, stop the process when the state is gone.
• References, not copies. A lua.Table, a lua.Value and a callable point to values in the registry of the guest state. They are valid while the state is valid. The garbage collector releases them.
• A state is fully isolated. The guest cannot see host globals or host modules. The host cannot see guest globals, except through the API. Two guest states never share values. A value of one state cannot go into the other state, and the bridge gives cannot pass <type> across independent states.
• Some results are discarded by design. state:eval and chunk:eval return the first result. chunk:call returns no result. Use chunk:pcall or fn:pcall to get all results.
• Errors are strings. An error in the guest becomes a plain string error on the host. The string is the guest message, and it can contain the guest position [string "..."]:line:. The library adds no position of its own to a guest error. An error that the library reports, such as an incorrect argument, contains the host position of the caller. Use pcall, chunk:pcall or fn:pcall to examine the error. Use chunk:xpcall when you need a guest traceback.
• Host callbacks exchange primitives and guest tables. An argument can be a primitive or a table. A table arrives as a live lua.Table view. A return value must be a primitive.
• A hook and the JIT engine interact. A hook needs interpreted code. Therefore, the installation of a hook disables the JIT engine of the guest until the removal of the hook.
• Set the debug names. Give chunkName to state:load(), or use chunk:setName. Use the prefix @ for a file path. Then a guest stack trace and debug.getinfo give a usable name.
• Bytecode is code. A chunk accepts bytecode by default, and bytecode runs with the full power of the guest. Examine data from a source that you do not trust with chunk:isBytecode(), or refuse it with chunk:setMode("text").

Bridge Design

lua-sys gives the Lua C API of LuaJIT to a guest lua_State that the host makes. This page explains two points:

• Why a compiled C bridge is necessary.
• How the re-entry limits control each design decision.

The Problem: Two Independent States

The call lua.new() makes a guest state with luaL_newstate(). This state is fully independent of the host LuaJIT interpreter. It has its own heap, its own global_State and its own call stack.

lua_xmove is the standard function for values between two states. However, that function operates on states that share a global_State only, such as coroutines or threads of the same root state. It does not operate here.

The only safe method between two independent states is a copy. The bridge reads a value from one state and pushes an equivalent value into the other state. This library copies primitives directly: nil, boolean, number and string. A compound type, such as a table, a function or a userdata, stays in the state that owns it. The other state uses a LUA_REGISTRYINDEX reference to it.

Why Not Use LuaJIT FFI for Calls?

The LuaJIT FFI lets Lua code call C functions and use C pointers. The lua-sys.raw module gives the complete Lua C API in this way. The functions raw.pcall, raw.rawgeti and raw.gettop are examples of FFI-bound functions.

FFI calls are sufficient for simple host to guest transitions. However, they stop the process under re-entry. The fault is in argv2cdata inside recff_cdata_call in the JIT recorder of LuaJIT.

What triggers the fault

The JIT recorder compiles a hot call site. If the call site has an FFI call with a lua_State* pointer, the recorder must make code that passes the pointer as a C argument. This step is argv2cdata. Under re-entry, the recorder finds the same FFI call while it records a trace for an outer call. The result is a fault.

This chain causes the fault:

Host Lua calls fn()                   <- JIT starts to record this call site
  fn() calls raw.pcall(guest_L, ...)  <- FFI call; JIT records argv2cdata for guest_L
    guest Lua runs
      guest calls host_callback()
        dispatch_callback (C) runs
          lua_pcall(host_L, ...)       <- runs host Lua in a C frame
            host Lua calls fn() again  <- JIT tries to record the same site again
              raw.pcall(guest_L, ...)  <- argv2cdata on guest_L during the trace
                                          => FAULT

The depth of the calls is not important. One condition is sufficient: a guest to host callback that causes the host to call a guest function through the FFI.

Why jit.off does not fully correct the problem

The mark jit.off on the FFI function prevents the JIT from compiling that function. It does not prevent the JIT from compiling the callers. A hot caller tries to record through the call boundary. The trace then inlines the FFI path, because the callee has no JIT metadata that identifies it as opaque.

In addition, jit.off makes each FFI call in that function interpreted. A call such as raw.pcall, raw.gettop or raw.settop costs approximately 1 to 8 ns when the JIT compiles it. The cost is higher when it is interpreted, and the bridge calls these functions at each cross-state transition.

The Solution: lua_CFunction Boundaries

A lua_CFunction is fully opaque to the JIT recorder. The JIT traces a call to a C function that lua_pushcfunction or lua_pushcclosure registered. It emits a call instruction and stops the trace. It never examines the function. This boundary is correct.

The bridge registers each cross-state call as a lua_CFunction:

Host to guest (bound_call): bridge.make_callable(guest_L_ptr, ref) pushes a bound_call closure onto the host state. The guest pointer and the function reference are C upvalues of the closure. Host code calls the closure as a normal Lua function. The JIT compiles the call site to bound_call and stops. Inside bound_call, the C code calls lua_pcall on the guest state. No FFI is necessary.

Guest to host (dispatch_callback): bridge.push_callback(guest_L_ptr, cb_id) pushes a dispatch_callback closure onto the guest state. Guest code that calls a host function dispatches through BC_FUNCC, which is a C function call and not an FFI call. Inside dispatch_callback, the C code calls lua_pcall on the host state.

In both directions, the transition is always:

Lua interpreter -> lua_CFunction (C) -> lua_pcall on the other state

The JIT never examines the other side of the boundary.

Callbacks That Make New States or Call the lua-sys API

A host callback from guest code runs inside the lua_pcall(host_L, ...) of dispatch_callback. At that moment, the JIT can be in the middle of a trace for the call site that started the guest code.

An FFI call from the host callback with a lua_State* cdata argument causes the argv2cdata conversion. Almost each raw.* function has this argument type. The result is the same fault as the first re-entry problem, but the host side causes it.

This condition occurs in the test runner of lde. The runner makes a new lua_State with lua.new() inside a host callback from a guest test runner state.

Correction: JIT engine off for each callback

dispatch_callback disables the JIT engine before the call lua_pcall(host_L, ...) and enables it again after the call. The enable step occurs on each path, including an error path:

luaJIT_setmode(host_L, 0, LUAJIT_MODE_ENGINE | LUAJIT_MODE_OFF);
int status = lua_pcall(host_L, nargs, LUA_MULTRET, 0);
luaJIT_setmode(host_L, 0, LUAJIT_MODE_ENGINE | LUAJIT_MODE_ON);

While the JIT is off, all code in the callback is interpreted. An FFI call is correct in interpreted code. The bridge prevents the JIT recording only, so argv2cdata never occurs. The JIT starts the compilation again when the callback returns.

The cost is that the body of a host callback is interpreted for its duration. A callback from a tight guest loop is therefore slower. Almost all lua-sys callbacks are short, for example to record a result or to make a state. Therefore, the cost is small.

Correction: bridge_new_state for safe state creation

The function lua.new() used raw.lnewstate() (FFI) and then raw.openlibs() (FFI). Both functions use a lua_State* cdata. The JIT-off correction makes this sequence safe from a callback. In addition, the bridge has bridge_new_state. This C function calls luaL_newstate() and luaL_openlibs() fully in C. It returns the pointer as a lightuserdata, not as cdata. The function lua.new() calls it and casts the result to lua_State* cdata on the host side:

local L = ffi.cast("lua_State*", bridge.new_state())

Thus state creation occurs behind a C boundary at each JIT state. This sequence agrees with the design rule that all cross-state operations go through lua_CFunction boundaries.

Debug Hooks

The method state:setHook installs a lua_Hook on the guest state. The hook dispatches to a host function, which is the same host and guest pattern as dispatch_callback.

A lua_Hook is a plain C function pointer without upvalues. Therefore, the bridge parks the callback reference in the guest registry under a private lightuserdata key, and the hook reads the key at each event:

hook fires (guest interpreter)
  hook_dispatch (C, lua_Hook)
    lua_getinfo(guest, "Sln", ar)     <- fill the debug fields in the guest
    build the info table on host_L (all debug fields, event, thread, shared mt)
    lua_pcall(host_L, ...)            <- run the host callback (JIT engine off)
    lua_error(guest) on a callback error <- stops guest code, catchable by pcall

The argument guest of the hook is the thread of the event. This thread is the main thread of the guest or a coroutine in it. hook_dispatch gives this thread to the host callback as info.thread, which is a lightuserdata of the lua_State*. Host code casts it back with ffi.cast("lua_State*", info.thread). Then it can call lua_getstack, lua_getinfo or lua_getlocal on the thread of the event. A stack trace or a local read on the main thread gives an incorrect result while a coroutine runs.

The bridge fills all debug fields (name, what, source, currentline and more) at the event. The info table also contains event, thread and a shared metatable. The metatable is made one time at module load and parked in the host registry, in the same way as the callable metatable. It supplies info:stack().

The method stack() examines the stack of the thread of the event with lua_getstack and lua_getinfo. It returns an array of lua.Frame objects. Item 1 is the frame of the hook event. The examination needs the thread paused at the hook. Therefore, the info table has a flag _hook_active. The bridge clears the flag when hook_dispatch finishes. A later call to stack() on a stored info table raises an error and does not examine an old stack.

A frame contains the thread and the level. Thus the host can do these steps:

• Read and write the active locals of the frame (frame:getLocal and frame:setLocal with lua_getlocal and lua_setlocal).
• Read and write the upvalues of its function (frame:getUpvalue and frame:setUpvalue).
• Evaluate code in the context of the frame (frame:eval).

The eval chunk runs in a new environment. The environment contains the locals and the upvalues of the frame. The metafields __index and __newindex connect it to the environment of the frame function (lua_getfenv). After the chunk runs, the bridge writes assignments to existing locals and upvalues back with lua_setlocal and lua_setupvalue. Thus frame:eval("x = 42") changes the running program.

A frame is valid while the callback runs only. Like the fields, a frame finds its lua.State through the guest registry, which all threads share. Therefore, a frame also operates on a coroutine frame.

LuaJIT fires hooks from the interpreter only. Code in a compiled trace does not dispatch through the hook. A hot while true do end becomes a LOOP bytecode, the JIT compiles it, and count hooks then stop without a message. Therefore, bridge_set_hook flushes the existing traces and disables the JIT engine of the guest while the hook is installed. bridge_remove_hook enables the engine again. The methods state:jitOff, state:jitOn and state:jitFlush give the same luaJIT_setmode calls for direct control.

Stack Safety Under Re-entry

The library supports host to guest to host chains. Therefore, the bridge can call dispatch_callback while bound_call runs on the C stack, and while the call stack of host_L is active. Both functions save and restore lua_gettop(host_L) around their work. Thus a nested call cannot corrupt the result slots of another call.

bound_call called from the host:
  guest_base = lua_gettop(guest)       <- save the guest stack
  lua_pcall(guest, ...)                <- guest runs, may call dispatch_callback
    dispatch_callback:
      saved_top = lua_gettop(host_L)   <- save the host stack depth
      lua_pcall(host_L, ...)           <- run the host callback
      lua_settop(host_L, saved_top)    <- restore the host stack on EACH path
  results start at guest_base+1
  lua_settop(guest, guest_base)        <- restore the guest stack

Without the call lua_settop(host_L, saved_top), each nested dispatch_callback leaves the host stack a small amount taller. After sufficient nesting, the stack overflows. A smaller problem is that the bridge reads the results of an inner call as the results of an outer call.

Value Passing: Why Only Primitives Cross Directly

The bridge does not copy a compound type, such as a table or a function. Such a value contains references to the GC heap of its owner state. A table from the guest state contains pointers into the memory of the guest. If the host uses these pointers after a guest GC cycle, the pointers can be invalid.

Therefore, a compound value stays in its home state and the other state uses a LUA_REGISTRYINDEX reference. A reference is a stable integer. It prevents the GC from collecting the value while the reference is live. The host receives a guest function as a makeCallable wrapper with a reference. The host receives a guest table as a lua.Table proxy that holds a reference.

Strings are the one exception. LuaJIT interns strings, and lua_tolstring returns a C const char* that is valid until the collection of the string. The bridge immediately calls lua_pushlstring into the destination state, which copies the bytes. Therefore, this step is safe.

Guest to Host Table Arguments

dispatch_callback has the same two paths as bound_call. If all arguments are primitives, it copies them directly. This is the fast path. If one argument is a table, it uses the slow path. Each table argument crosses as a pair (tag, ref):

• A lightuserdata tag from bridge.compound_tag().
• A LUA_REGISTRYINDEX reference that the bridge takes in the guest.

The host helper dispatchCallbackSlow is registered one time and is carried in upvalue 2 of the closure. It converts each pair back into a lua.Table proxy. For this step, it finds the owner lua.State in a map with lightuserdata keys that lua.new() fills. Then it calls the real callback with the original argument order and the nil slots.

The proxies are live views. A read or a write goes directly to the guest table. The bridge releases the guest reference at the collection of the proxy.

All other compound argument types, such as functions, userdata and threads, give the error cannot pass. A return value from the host to the guest must be a primitive.

Performance Notes

Each cross-state transition costs a minimum of lua_rawgeti and lua_pcall on the destination state. On a modern CPU, this cost is approximately 18 ns.

The C bridge adds this overhead for each call:

The callback lookup in dispatch_callback uses a cached luaL_ref integer, which is an O(1) lua_rawgeti. It does not use lua_getfield on the registry string key, which is an O(n) hash lookup. Thus each guest to host call saves approximately 13 ns.