Introducing luzer, a coverage-guided Lua fuzzing engine
luzer is heavily inspired by Google’s Atheris, a Python fuzzing
engine. Like Atheris, luzer uses
libFuzzer for its coverage instrumentation and fuzzing engine.
luzer also supports AddressSanitizer and
UndefinedBehaviorSanitizer
when fuzzing Lua C libraries.
This post will go over a motivation behind building luzer, provide a brief overview of installing and running the tool, and discuss some of its interesting implementation details.
Bringing fuzzing testing to Lua
The Fuzzing Book provides the following definition of fuzzing:
Fuzzing represents a dynamic testing method that inputs malformed or unpredictable data to a system to detect security issues, bugs, or system failures. We consider it an essential tool to include in your testing suite.
Fuzzing is an important testing methodology when developing high-assurance software, even in Lua. Consider AFL’s extensive trophy case, and OSS-Fuzz’s claim that it’s helped find and fix over 10.000 security vulnerabilities and 36.000 bugs with fuzzing. As mentioned previously, Python has Atheris. Java has Jazzer. Ruby has Ruzzy. The Lua community deserves a high-quality, modern fuzzing tool too.
This isn’t to say that Lua fuzzers haven’t been built before.
They have: afl-lua,
a coverage-guided AFL-based fuzzing engine, and
lua-quickcheck,
a blackbox property-based library. However, all these tools appear
to be either unmaintained, difficult to use, lacking features, or
all of the above. Moreover, afl-lua supports PUC Rio Lua only and
require a modified Lua runtime. To address these challenges, luzer
is built on three principles:
- Fuzz Lua code and Lua C libraries.
- Make fuzzing easy by providing a luarocks installation process and simple interface.
- Integrate with the extensive libFuzzer and AFL ecosystem.
With that, let’s give this thing a test drive.
Installing and running luzer
The luzer repository is well
documented,
so this post will provide an abridged version of installing and
running the tool. The goal here is to provide a quick overview of
what using luzer looks like. For more information, check out the
repository.
First things first, luzer requires a Linux environment and a recent version of Clang (I’ve tested back to version 15.0.0). Releases of Clang can be found on its GitHub releases page. With that out of the way, let’s get started.
Run the following command to install luzer using luarocks:
$ luarocks --local install luzer
$ eval $(luarocks path)
To make luarocks-installed modules available in your shell, you
should run eval $(luarocks path).
Fuzzing Lua code
Well, let’s demonstrate a fuzzing of Lua code. For simplicity,
we combine a target code and test code into a single function.
We have a simple fuzzing target that crashes when it receives the
input started with “L”. It’s a contrived example, but it demonstrates luzer’s
pass conditions and produce crashes. Let’s call a file with test harness oops.lua:
local luzer = require("luzer")
local function TestOneInput(buf)
local fdp = luzer.FuzzedDataProvider(buf)
local str = fdp:consume_string(100)
if str == "L" then
assert(nil, "assert has triggered")
end
return
end
luzer.Fuzz(TestOneInput)
You can start the fuzzing process with the following command:
$ lua5.1 examples/example_basic.lua -print_pcs=1
This should quickly produce a crash like the following:
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 473228111
INFO: Loaded 1 modules (77 inline 8-bit counters): 77 [0x7e8f5eee6963, 0x7e8f5eee69b0),
INFO: Loaded 1 PC tables (77 PCs): 77 [0x7e8f5eee69b0,0x7e8f5eee6e80),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2 INITED cov: 17 ft: 18 corp: 1/1b exec/s: 0 rss: 28Mb
#7 NEW cov: 17 ft: 28 corp: 2/3b lim: 4 exec/s: 0 rss: 28Mb L: 2/2 MS: 5 ChangeByte-CrossOver-ChangeByte-ChangeBit-InsertByte-
#11 NEW cov: 17 ft: 38 corp: 3/7b lim: 4 exec/s: 0 rss: 28Mb L: 4/4 MS: 4 ChangeByte-CopyPart-CrossOver-CrossOver-
NEW_PC: 0x7e8f5eea071f
#15 NEW cov: 18 ft: 39 corp: 4/9b lim: 4 exec/s: 0 rss: 28Mb L: 2/4 MS: 4 ChangeBinInt-ChangeBinInt-ShuffleBytes-ChangeByte-
#23 NEW cov: 18 ft: 47 corp: 5/12b lim: 4 exec/s: 0 rss: 28Mb L: 3/4 MS: 3 ShuffleBytes-ChangeBinInt-InsertByte-
#29 NEW cov: 18 ft: 48 corp: 6/14b lim: 4 exec/s: 0 rss: 28Mb L: 2/4 MS: 1 CrossOver-
NEW_PC: 0x7e8f5eea04c3
#32 NEW cov: 19 ft: 49 corp: 7/15b lim: 4 exec/s: 0 rss: 28Mb L: 1/4 MS: 3 ShuffleBytes-ChangeBit-ChangeByte-
#40 NEW cov: 19 ft: 50 corp: 8/18b lim: 4 exec/s: 0 rss: 28Mb L: 3/4 MS: 3 ShuffleBytes-CrossOver-CrossOver-
luajit: luzer/tests/test_e2e.lua:7: assert has triggered
stack traceback:
[C]: in function 'assert'
luzer/tests/test_e2e.lua:7: in function <luzer/tests/test_e2e.lua:3>
[C]: in function 'Fuzz'
./luzer/init.lua:69: in function 'Fuzz'
luzer/tests/test_e2e.lua:15: in main chunk
[C]: at 0x64d5e9a250c0
==87538== ERROR: libFuzzer: fuzz target exited
SUMMARY: libFuzzer: fuzz target exited
MS: 1 ChangeBit-; base unit: 08534f33c201a45017b502e90a800f1b708ebcb3
0x4c,
L
artifact_prefix='./'; Test unit written to ./crash-d160e0986aca4714714a16f29ec605af90be704d
Base64: TA==
luzer used libFuzzer’s coverage-guided instrumentation to discover the input that produces a crash. This is one of luzer’s key contributions: coverage-guided support for Lua code. We will discuss coverage support and more in the next section.
Fuzzing Lua C libraries
To facilitate testing the tool, let’s consider a small example with Lua C extension and an assertion that can be triggered under condition. This section will demonstrate using luzer to fuzz a C extension with a reachable assertion:
#include <string.h>
#include <assert.h>
#include "lua.h"
#include "lauxlib.h"
static int
lua_say_hello(lua_State *L) {
size_t len;
const char *str = luaL_checklstring(L, 1, &len);
char msg[] = "Hello, Lua!";
if (strncmp(str, msg, sizeof(msg)) == 0) {
assert(NULL);
}
lua_pop(L, 1);
return 0;
}
static const struct luaL_Reg functions [] = {
{ "say_hello", lua_say_hello },
{ NULL, NULL }
};
int luaopen_hello(lua_State *L) {
#if LUA_VERSION_NUM == 501
luaL_register(L, "hello", functions);
#else
luaL_newlib(L, functions);
#endif
return 1;
}
Compile a C code to a shared library:
clang hello.c -shared -o hello.so -Llua5.1 -I/usr/include/lua5.1/ -fsanitize=fuzzer-no-link
The library will crash when passed string is equal to “Hello, Lua!”:
$ lua5.1 -e 'require("hello").say_hello("Hello, Lua!")'
lua5.1: hello.c:15: int lua_say_hello(lua_State *): Assertion `NULL' failed.
Aborted (core dumped)
Next we create a test harness hello_test.lua:
local luzer = require("luzer")
local hello = require("hello")
local function TestOneInput(buf)
hello.say_hello(buf)
end
local args = {
print_pcs = 1,
}
luzer.Fuzz(TestOneInput, nil, args)
And execute test harness:
$ LD_PRELOAD=$(lua5.1 -e "print(require('luzer').path.asan)") lua5.1 hello_test.lua -runs=1000000
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1943943233
INFO: Loaded 2 modules (82 inline 8-bit counters): 77 [0x7c3204bca963, 0x7c3204bca9b0), 5 [0x7c3205b39048, 0x7c3205b3904d),
INFO: Loaded 2 PC tables (82 PCs): 77 [0x7c3204bca9b0,0x7c3204bcae80), 5 [0x7c3205b39050,0x7c3205b390a0),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2 INITED cov: 2 ft: 3 corp: 1/1b exec/s: 0 rss: 34Mb
luajit: hello.c:13: int lua_say_hello(lua_State *): Assertion `NULL' failed.
==123223== ERROR: libFuzzer: deadly signal
#0 0x7c3205124331 in __sanitizer_print_stack_trace (/home/sergeyb/sources/luzer/build/luzer/libfuzzer_with_asan.so+0x124331) (BuildId: 0b1ec97d217c690abc36df6da4aa0e0d7ef71320)
#1 0x7c32050757d8 in fuzzer::PrintStackTrace() (/home/sergeyb/sources/luzer/build/luzer/libfuzzer_with_asan.so+0x757d8) (BuildId: 0b1ec97d217c690abc36df6da4aa0e0d7ef71320)
#2 0x7c320505b0e3 in fuzzer::Fuzzer::CrashCallback() (/home/sergeyb/sources/luzer/build/luzer/libfuzzer_with_asan.so+0x5b0e3) (BuildId: 0b1ec97d217c690abc36df6da4aa0e0d7ef71320)
#3 0x7c3204c4532f (/lib/x86_64-linux-gnu/libc.so.6+0x4532f) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
#4 0x7c3204c9eb2b in __pthread_kill_implementation nptl/pthread_kill.c:43:17
#5 0x7c3204c9eb2b in __pthread_kill_internal nptl/pthread_kill.c:78:10
#6 0x7c3204c9eb2b in pthread_kill nptl/pthread_kill.c:89:10
#7 0x7c3204c4527d in raise signal/../sysdeps/posix/raise.c:26:13
#8 0x7c3204c288fe in abort stdlib/abort.c:79:7
#9 0x7c3204c2881a in __assert_fail_base assert/assert.c:96:3
#10 0x7c3204c3b516 in __assert_fail assert/assert.c:105:3
#11 0x7c3205b362ce in lua_say_hello hello.c
#12 0x642f716995d5 (/usr/bin/luajit+0x785d5) (BuildId: e8ba96877bda295db49e40559a6a7038587195fd)
#13 0x7c3204b82a19 in luaL_test_one_input /home/sergeyb/sources/luzer/luzer/luzer.c:287:2
#14 0x7c3204b82961 in TestOneInput /home/sergeyb/sources/luzer/luzer/luzer.c:342:11
#15 0x7c320505c78b in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/sergeyb/sources/luzer/build/luzer/libfuzzer_with_asan.so+0x5c78b) (BuildId: 0b1ec97d217c690abc36df6da4aa0e0d7ef71320)
#16 0x7c320505be25 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/home/sergeyb/sources/luzer/build/luzer/libfuzzer_with_asan.so+0x5be25) (BuildId: 0b1ec97d217c690abc36df6da4aa0e0d7ef71320)
#17 0x7c320505d7d5 in fuzzer::Fuzzer::MutateAndTestOne() (/home/sergeyb/sources/luzer/build/luzer/libfuzzer_with_asan.so+0x5d7d5) (BuildId: 0b1ec97d217c690abc36df6da4aa0e0d7ef71320)
#18 0x7c320505e2a5 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/home/sergeyb/sources/luzer/build/luzer/libfuzzer_with_asan.so+0x5e2a5) (BuildId: 0b1ec97d217c690abc36df6da4aa0e0d7ef71320)
#19 0x7c320504b6b5 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/sergeyb/sources/luzer/build/luzer/libfuzzer_with_asan.so+0x4b6b5) (BuildId: 0b1ec97d217c690abc36df6da4aa0e0d7ef71320)
#20 0x7c3204b8307a in luaL_fuzz /home/sergeyb/sources/luzer/luzer/luzer.c:537:11
#21 0x642f716995d5 (/usr/bin/luajit+0x785d5) (BuildId: e8ba96877bda295db49e40559a6a7038587195fd)
#22 0x642f7163bc8f in lua_pcall (/usr/bin/luajit+0x1ac8f) (BuildId: e8ba96877bda295db49e40559a6a7038587195fd)
#23 0x642f7163315b (/usr/bin/luajit+0x1215b) (BuildId: e8ba96877bda295db49e40559a6a7038587195fd)
#24 0x642f71634a06 (/usr/bin/luajit+0x13a06) (BuildId: e8ba96877bda295db49e40559a6a7038587195fd)
#25 0x642f716995d5 (/usr/bin/luajit+0x785d5) (BuildId: e8ba96877bda295db49e40559a6a7038587195fd)
#26 0x642f7163bce3 in lua_cpcall (/usr/bin/luajit+0x1ace3) (BuildId: e8ba96877bda295db49e40559a6a7038587195fd)
#27 0x642f716296d6 in main (/usr/bin/luajit+0x86d6) (BuildId: e8ba96877bda295db49e40559a6a7038587195fd)
#28 0x7c3204c2a1c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#29 0x7c3204c2a28a in __libc_start_main csu/../csu/libc-start.c:360:3
#30 0x642f71629754 in _start (/usr/bin/luajit+0x8754) (BuildId: e8ba96877bda295db49e40559a6a7038587195fd)
NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 3 CopyPart-ChangeBinInt-CMP- DE: "Hello, Lua!"-; base unit: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
0x48,0x65,0x6c,0x6c,0x6f,0x2c,0x20,0x4c,0x75,0x61,0x21,0x0,
Hello, Lua!\000
artifact_prefix='./'; Test unit written to ./crash-513aeb2875999bcdc4b044defb8ec98663cb946b
Base64: SGVsbG8sIEx1YSEA
The previous example has demonstrated a test for our own Lua C library. The following example will demostrate an example with third-party extension that use Lua C library that we’d like to fuzz.
To fuzz a Lua C extension, we need a way to compile the extension with libFuzzer and its associated sanitizers. Compiling C/C++ code for fuzzing requires special compile-time flags, so we need a way to inject these flags into the C extension compilation process. Dynamically adding these flags is important because we’d like to install and fuzz Lua libraries without having to modify the underlying code.
We can use the following command to install Lua rocks containing C extensions with the appropriate fuzzing flags:
luarocks install --tree modules --lua-version 5.1 lua-cmsgpack 0.4.0-0 CC="clang" CFLAGS="-ggdb -fPIC -fsanitize=fuzzer-no-link"
Installing https://luarocks.org/lua-cmsgpack-0.4.0-0.rockspec
Cloning into 'lua-cmsgpack'...
remote: Enumerating objects: 257, done.
remote: Total 257 (delta 0), reused 0 (delta 0), pack-reused 257 (from 1)
Receiving objects: 100% (257/257), 99.66 KiB | 843.00 KiB/s, done.
Resolving deltas: 100% (129/129), done.
Note: switching to 'dec1810a70d2948725f2e32cc38163de62b9d9a7'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
lua-cmsgpack 0.4.0-0 depends on lua >= 5.1 (5.1-1 provided by VM)
clang -ggdb -fPIC -fsanitize=fuzzer-no-link -I/usr/include/lua5.1/ -c lua_cmsgpack.c -o lua_cmsgpack.o
gcc -shared -o cmsgpack.so lua_cmsgpack.o
No existing manifest. Attempting to rebuild...
lua-cmsgpack 0.4.0-0 is now installed in /home/sergeyb/sources/bronevichok.ru/modules (license: Two-clause BSD)
$ eval $(luarocks path)
$ export LUA_CPATH="$LUA_CPATH;modules/lib/lua/5.1/?.so;../../?.so"
$ export LUA_PATH="$LUA_PATH;modules/lib/lua/5.1/?.lua"
The following Lua code contains a test harness for cmsgpack library:
local luzer = require("luzer")
local msgpack = require("cmsgpack")
local msgpack_safe = require("cmsgpack.safe")
local unpack = unpack or table.unpack
local function TestOneInput(buf)
if #buf == 0 then return end
local data, err = msgpack_safe.unpack(buf)
if not err and data ~= 0 then
local res = { msgpack_safe.unpack(data) }
if #res ~= 0 then
msgpack.pack(unpack(res))
end
end
end
luzer.Fuzz(TestOneInput)
In this example, the msgpack.pack() and msgpack.unpack() functions
are passed to luzer.Fuzz(). First, luzer calls a function LLVMFuzzerRunDriver
with a function pointer. Then, every time that
function pointer is invoked, it calls TestOneInput() to execute the Lua target.
This allows the C/C++ fuzzing engine to repeatedly call the Lua target with
fuzzed data. Considering the example above, this accomplishes the
goal of calling arbitrary Lua code from a C/C++ library.
Let’s start fuzzing using test harness for cmsgpack:
LD_PRELOAD=build/luzer/libfuzzer_with_asan.so tarantool msgpack.lua
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1177135163
INFO: Loaded 2 modules (243 inline 8-bit counters): 77 [0x7ffff5eca963, 0x7ffff5eca9b0), 166 [0x7ffff668d310, 0x7ffff668d3b6),
INFO: Loaded 2 PC tables (243 PCs): 77 [0x7ffff5eca9b0,0x7ffff5ecae80), 166 [0x7ffff668d3b8,0x7ffff668de18),
[New Thread 0x7bffeecb96c0 (LWP 38743)]
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2 INITED cov: 30 ft: 31 corp: 1/1b exec/s: 0 rss: 188Mb
#3 NEW cov: 30 ft: 46 corp: 2/2b lim: 4 exec/s: 0 rss: 188Mb L: 1/1 MS: 1 ChangeByte-
#4 NEW cov: 30 ft: 60 corp: 3/4b lim: 4 exec/s: 0 rss: 188Mb L: 2/2 MS: 1 InsertByte-
#20 NEW cov: 30 ft: 70 corp: 4/5b lim: 4 exec/s: 0 rss: 188Mb L: 1/2 MS: 1 EraseBytes-
#22 NEW cov: 33 ft: 75 corp: 5/8b lim: 4 exec/s: 0 rss: 188Mb L: 3/3 MS: 2 CrossOver-InsertByte-
#25 NEW cov: 35 ft: 77 corp: 6/9b lim: 4 exec/s: 0 rss: 188Mb L: 1/3 MS: 3 InsertByte-EraseBytes-ChangeByte-
#32 NEW cov: 36 ft: 79 corp: 7/11b lim: 4 exec/s: 0 rss: 188Mb L: 2/3 MS: 2 CrossOver-InsertByte-
NEW_FUNC[1/1]: [Detaching after fork from child process 38744]
0x7ffff6689450 in mp_decode_to_lua_hash /tmp/luarocks_lua-cmsgpack-0.4.0-0-5478545/lua-cmsgpack/lua_cmsgpack.c:561
<snipped>
#254022 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 841 exec/s: 127011 rss: 188Mb L: 89/809 MS: 3 InsertByte-ChangeBinInt-EraseBytes-
#255673 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 850 exec/s: 127836 rss: 188Mb L: 15/809 MS: 1 EraseBytes-
#259576 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 886 exec/s: 129788 rss: 188Mb L: 94/809 MS: 3 ShuffleBytes-InsertByte-EraseBytes-
#259928 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 886 exec/s: 129964 rss: 188Mb L: 318/809 MS: 2 PersAutoDict-EraseBytes- DE: "\031\00
0\000\000"-
#260186 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 886 exec/s: 130093 rss: 188Mb L: 36/809 MS: 3 ShuffleBytes-CopyPart-EraseBytes-
#262144 pulse cov: 137 ft: 689 corp: 382/20Kb lim: 904 exec/s: 131072 rss: 188Mb
#263653 NEW cov: 137 ft: 690 corp: 383/21Kb lim: 913 exec/s: 131826 rss: 188Mb L: 887/887 MS: 2 CopyPart-CopyPart-
#265109 REDUCE cov: 137 ft: 690 corp: 383/21Kb lim: 922 exec/s: 132554 rss: 188Mb L: 25/887 MS: 1 EraseBytes-
#265187 REDUCE cov: 137 ft: 690 corp: 383/21Kb lim: 922 exec/s: 132593 rss: 188Mb L: 24/887 MS: 3 ChangeBit-ShuffleBytes-EraseBytes-
#267424 REDUCE cov: 137 ft: 690 corp: 383/21Kb lim: 940 exec/s: 133712 rss: 188Mb L: 38/887 MS: 2 CopyPart-EraseBytes-
#267515 REDUCE cov: 137 ft: 690 corp: 383/21Kb lim: 940 exec/s: 133757 rss: 188Mb L: 65/887 MS: 1 EraseBytes-
#267753 NEW cov: 137 ft: 691 corp: 384/22Kb lim: 940 exec/s: 133876 rss: 188Mb L: 937/937 MS: 3 ChangeByte-ChangeByte-CopyPart-
#268845 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 949 exec/s: 134422 rss: 188Mb L: 37/937 MS: 2 CopyPart-EraseBytes-
#270625 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 958 exec/s: 135312 rss: 188Mb L: 145/937 MS: 5 CopyPart-CopyPart-PersAutoDict-EraseB
ytes-InsertByte- DE: "\377\377"-
#274896 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 994 exec/s: 137448 rss: 188Mb L: 10/937 MS: 1 EraseBytes-
#276032 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 1003 exec/s: 138016 rss: 188Mb L: 36/937 MS: 1 EraseBytes-
#276065 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 1003 exec/s: 138032 rss: 188Mb L: 34/937 MS: 3 ChangeBit-ChangeBinInt-EraseBytes-
#277030 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 1012 exec/s: 138515 rss: 188Mb L: 936/936 MS: 5 EraseBytes-ChangeBinInt-InsertByte-I
nsertRepeatedBytes-CopyPart-
#277081 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 1012 exec/s: 138540 rss: 188Mb L: 17/936 MS: 1 EraseBytes-
#278252 REDUCE cov: 137 ft: 693 corp: 385/22Kb lim: 1021 exec/s: 139126 rss: 188Mb L: 35/936 MS: 1 CrossOver-
#279465 NEW cov: 137 ft: 699 corp: 386/22Kb lim: 1030 exec/s: 139732 rss: 188Mb L: 35/936 MS: 3 CrossOver-ChangeBinInt-ChangeBinInt-
#280317 REDUCE cov: 137 ft: 699 corp: 386/22Kb lim: 1030 exec/s: 140158 rss: 188Mb L: 173/936 MS: 2 ChangeBit-EraseBytes-
#282109 REDUCE cov: 137 ft: 699 corp: 386/22Kb lim: 1040 exec/s: 141054 rss: 188Mb L: 213/936 MS: 2 PersAutoDict-EraseBytes- DE: "\000\0
00\000\000\000\000\000\000"-
#282306 NEW cov: 137 ft: 700 corp: 387/23Kb lim: 1040 exec/s: 141153 rss: 188Mb L: 1021/1021 MS: 2 InsertByte-CopyPart-
#283698 REDUCE cov: 137 ft: 700 corp: 387/23Kb lim: 1050 exec/s: 141849 rss: 188Mb L: 13/1021 MS: 2 InsertByte-EraseBytes-
#285356 REDUCE cov: 137 ft: 700 corp: 387/23Kb lim: 1060 exec/s: 142678 rss: 188Mb L: 391/1021 MS: 3 ChangeBinInt-ChangeByte-EraseBytes-
Thread 1 "tarantool" received signal SIGSEGV, Segmentation fault.
0x00007ffff6688098 in mp_decode_to_lua_array (L=0x7bfff07021b0, c=0x0, len=93824994521398) at lua_cmsgpack.c:548
warning: 548 lua_cmsgpack.c: No such file or directory
Oh, no, the fuzzing process was interrupted by segmentation fault.
Probably it’s because msgpack buffer with a huge number of nested
arrays could lead to a stack overflow in cmsgpack.
Interesting implementation details
You don’t need to understand this section to use luzer, but fuzzing can often be more art than science, so we wanted to share some details to help demystify this dark art. We certainly learned a lot from the blog posts describing Atheris and Jazzer, so we figured we’d pay it forward. Of course, there are many interesting details that go into creating a tool like this but we’ll focus on three: creating a Lua fuzzing harness, compiling Lua C extensions with libFuzzer, and adding coverage support for Lua code.
Creating a Lua fuzzing harness
One of the first things you need when embarking on a fuzzing campaign is a fuzzing harness. The Fuzzing Book defines a fuzzing harness as follows:
A harness handles the test setup for a given target. The harness wraps the software and initializes it such that it is ready for executing test cases. A harness integrates a target into a testing environment.
When fuzzing Lua code, naturally we want to write our fuzzing
harness in Lua, too. This speaks to goal number two from the
beginning of this post: make fuzzing Lua simple and easy. However,
a problem arises when we consider that libFuzzer is written in C/C++.
When using libFuzzer as a library, we need to pass a C function
pointer to LLVMFuzzerRunDriver to initiate the fuzzing process.
How can we pass arbitrary Lua code to a C/C++ library?
Using a foreign function interface (FFI) like Lua-FFI is one possibility. However, FFIs are generally used to go the other direction: calling C/C++ code from Lua. Lua C API seem like another possibility, but we still need to figure out a way to pass arbitrary Lua code to a C extension. This function allowed us to use Lua C extensions to bridge the gap between Lua code and the libFuzzer C/C++ implementation. Perfect, this is exactly what we needed.
Adding coverage support for Lua code
Instead of Lua C extensions, what if we want to fuzz Lua code? That is, Lua projects that written in Lua language without Lua C libraries.
First, we need to cover the motivation for coverage support. Fuzzers derive some of their “smarts” from analyzing coverage information. This is a lot like code coverage information provided by unit and integration tests. While fuzzing, most fuzzers prioritize inputs that unlock new code branches. This increases the likelihood that they will find crashes and bugs. When fuzzing Lua C extensions, luzer can punt coverage instrumentation for C code to Clang. With Lua code, we have no such luxury.
While implementing luzer, I’ve discovered one supremely useful piece of
functionality: the hook mechanism in a Lua’s debug module. The debug
library is the official Lua API for listening for certain types of events like
calling a function, returning from a function, executing a line of code, and
execution every count instructions. When these events fire, you can execute a
callback function to handle the event however you’d like. So, this sounds
great, however there is no possibility to hook branch events and all that
remains is to use “line” event. This is a serios limitation in fuzzing Lua code and
I’ve sent a mail with proposal
to introduce a new event in a Lua hook mechanism. afl-lua workarounds
this limitation by modification in PUC Rio Lua source code. This works fine,
but I don’t like this solution. There are five major releases of
PUC Rio Lua (5.1, 5.2, 5.3, 5.4 and 5.5) that are not binary compatible with each other
and also don’t forget about another popular Lua runtime - LuaJIT. So for supporting
al these runtimes one should support patches with branch event implementation.
So the current implementation of luzer uses a hook for “line” that feed a
data obtained by callback function into libFuzzer in real time and it will fuzz
accordingly. Discussing how to feed this data into libFuzzer is beyond the
scope of this post, but if you’d like to learn more, we use SanitizerCoverage’s
inline 8-bit
counters,
PC-Table, and
data flow
tracing.
Having coverage events included in the standard library can make our lives a lot easier. Without it, we may have had to resort to much more invasive and cumbersome solutions like modifying the Lua code the interpreter sees in real time. However, it would have made our lives even easier if hooking coverage events were part of the official, public Lua C API.
Find more Lua and C bugs with luzer
We faced some interesting challenges while building this tool and attempted to hide much of the complexity behind a simple, easy to use interface. When using the tool, the implementation details should not become a hindrance or an annoyance. However, discussing them here in detail may spur the next fuzzer implementation or step forward in the fuzzing community. As mentioned previously, the Atheris and Jazzer posts were a great inspiration to us, so we figured we’d pay it forward.
Building the tool is just the beginning. The real value comes when
we start using the tool to find bugs. Like Atheris for Python,
Jazzer for Java, and Ruzzy for Ruby before it, luzer is an
attempt to bring a higher level of software assurance to the Lua
community. If you find a bug using luzer, feel free to open a PR
against our trophy case with a
link to the issue.
If you’d like to read more about our work on fuzzing, check out the following posts:
- (In Russian) Доклад: Добавляем поддержку скриптового языка в AFL и LibFuzzer на примере Lua: видео, слайды
- (In Russian) Фаззинг как основа эффективной разработки на примере LuaJIT: видео, слайды