Commits
- Commit:
fffd8298be288a0d5d046aa46882587a09bacf14
- From:
- Sergey Bronnikov <sergeyb@tarantool.org>
- Date:
test/fuzz: speedup string serialization
- clamp before cleaning string because cleaning is not cheap
(O(n), where max n is equal to kMaxStrLength)
- call cleaning for identifiers only, there is no sense to
cleaning string literals
- replace symbols disallowed by Lua grammar in indentifier's
names with '_'
The patch saves 16 sec on 145k samples (401 sec before the patch
and 385 sec after the patch). It is actually not so much, but it
is about 2.5 min per hour.
NO_CHANGELOG=testing
NO_DOC=testing
- Commit:
9d3859b246977c307a22f5930755fe77e374a634
- From:
- Vladimir Davydov <vdavydov@tarantool.org>
- Via:
- Vladimir Davydov <vdavydov.dev@gmail.com>
- Date:
vinyl: fix gc vs vylog race leading to duplicate record
Vinyl run files aren't always deleted immediately after compaction,
because we need to keep run files corresponding to checkpoints for
backups. Such run files are deleted by the garbage collection procedure,
which performs the following steps:
1. Loads information about all run files from the last vylog file.
2. For each loaded run record that is marked as dropped:
a. Tries to remove the run files.
b. On success, writes a "forget" record for the dropped run,
which will make vylog purge the run record on the next
vylog rotation (checkpoint).
(see `vinyl_engine_collect_garbage()`)
The garbage collection procedure writes the "forget" records
asynchronously using `vy_log_tx_try_commit()`, see `vy_gc_run()`.
This procedure can be successfully executed during vylog rotation,
because it doesn't take the vylog latch. It simply appends records
to a memory buffer which is flushed either on the next synchronous
vylog write or vylog recovery.
The problem is that the garbage collection isn't necessarily loads
the latest vylog file because the vylog file may be rotated between
it calls `vy_log_signature()` and `vy_recovery_new()`. This may
result in a "forget" record written twice to the same vylog file
for the same run file, as follows:
1. GC loads last vylog N
2. GC starts removing dropped run files.
3. CHECKPOINT starts vylog rotation.
4. CHECKPOINT loads vylog N.
5. GC writes a "forget" record for run A to the buffer.
6. GC is completed.
7. GC is restarted.
8. GC finds that the last vylog is N and blocks on the vylog latch
trying to load it.
9. CHECKPOINT saves vylog M (M > N).
10. GC loads vylog N. This triggers flushing the forget record for
run A to vylog M (not to vylog N), because vylog M is the last
vylog at this point of time.
11. GC starts removing dropped run files.
12. GC writes a "forget" record for run A to the buffer again,
because in vylog N it's still marked as dropped and not forgotten.
(The previous "forget" record was written to vylog M).
13. Now we have two "forget" records for run A in vylog M.
Such duplicate run records aren't tolerated by the vylog recovery
procedure, resulting in a permanent error on the next checkpoint:
```
ER_INVALID_VYLOG_FILE: Invalid VYLOG file: Run XXXX forgotten but not registered
```
To fix this issue, we move `vy_log_signature()` under the vylog latch
to `vy_recovery_new()`. This makes sure that GC will see vylog records
that it's written during the previous execution.
Catching this race in a function test would require a bunch of ugly
error injections so let's assume that it'll be tested by fuzzing.
Closes #10128
NO_DOC=bug fix
NO_TEST=tested manually with fuzzer
- Commit:
19d1f1cc0a2180730219a5fe24580c6d7fbcc239
- From:
- Vladimir Davydov <vdavydov@tarantool.org>
- Via:
- Vladimir Davydov <vdavydov.dev@gmail.com>
- Date:
tuple: don't use offset_slot_cache in vinyl threads
`key_part::offset_slot_cache` and `key_part::format_epoch` are used for
speeding up tuple field lookup in `tuple_field_raw_by_part()`. These
structure members are accessed and updated without any locks, assuming
this code is executed exclusively in the tx thread. However, this isn't
necessarily true because we also perform tuple field lookups in vinyl
read threads. Apparently, this can result in unexpected races and bugs,
for example:
```
#1 0x590be9f7eb6d in crash_collect+256
#2 0x590be9f7f5a9 in crash_signal_cb+100
#3 0x72b111642520 in __sigaction+80
#4 0x590bea385e3c in load_u32+35
#5 0x590bea231eba in field_map_get_offset+46
#6 0x590bea23242a in tuple_field_raw_by_path+417
#7 0x590bea23282b in tuple_field_raw_by_part+203
#8 0x590bea23288c in tuple_field_by_part+91
#9 0x590bea24cd2d in unsigned long tuple_hint<(field_type)5, false, false>(tuple*, key_def*)+103
#10 0x590be9d4fba3 in tuple_hint+40
#11 0x590be9d50acf in vy_stmt_hint+178
#12 0x590be9d53531 in vy_page_stmt+168
#13 0x590be9d535ea in vy_page_find_key+142
#14 0x590be9d545e6 in vy_page_read_cb+210
#15 0x590be9f94ef0 in cbus_call_perform+44
#16 0x590be9f94eae in cmsg_deliver+52
#17 0x590be9f9583e in cbus_process+100
#18 0x590be9f958a5 in cbus_loop+28
#19 0x590be9d512da in vy_run_reader_f+381
#20 0x590be9cb4147 in fiber_cxx_invoke(int (*)(__va_list_tag*), __va_list_tag*)+34
#21 0x590be9f8b697 in fiber_loop+219
#22 0x590bea374bb6 in coro_init+120
```
Fix this by skipping this optimization for threads other than tx.
No test is added because reproducing this race is tricky. Ideally, bugs
like this one should be caught by fuzzing tests or thread sanitizers.
Closes #10123
NO_DOC=bug fix
NO_TEST=tested manually with fuzzer
- Commit:
7b72080ddf926d271a4a4cad6f6ffe6e6c00e332
- From:
- Vladimir Davydov <vdavydov@tarantool.org>
- Via:
- Vladimir Davydov <vdavydov.dev@gmail.com>
- Date:
vinyl: fix cache iterator skipping tuples in read view
The tuple cache doesn't store older tuple versions so if a reader is
in a read view, it must skip tuples that are newer than the read view,
see `vy_cache_iterator_stmt_is_visible()`. A reader must also ignore
cached intervals if any of the tuples used as a boundary is invisible
from the read view, see `vy_cache_iterator_skip_to_read_view()`.
There's a bug in `vy_cache_iterator_restore()` because of which such
an interval may be returned to the reader: when we step backwards
from the last returned tuple we consider only one of the boundaries.
As a result, if the other boundary is invisible from the read view,
the reader will assume there's nothing in the index between the
boundaries and skip reading older sources (memory, disk). Fix this by
always checking if the other boundary is visible.
Closes #10109
NO_DOC=bug fix
- Commit:
72763f947b94a99ccc8bb5e0fff2cce943b38497
- From:
- Vladimir Davydov <vdavydov@tarantool.org>
- Via:
- Vladimir Davydov <vdavydov.dev@gmail.com>
- Date:
vinyl: fix run iterator skipping tuples following non-terminal statement
If a run iterator is positioned at a non-terminal statement (UPSERT or
UPDATE), `vy_run_iterator_next()` will iterate over older statements
with the same key using `vy_run_iterator_next_lsn()` to build the key
history. While doing so, it may reach the end of the run file (if the
current key is the last in the run). This would stop iteration
permanently, which is apparently wrong for reverse iterators (LE or LT):
if this happens the run iterator won't return any keys preceding the
last one in the run file. Fix this by removing `vy_run_iterator_stop()`
from `vy_run_iterator_next_lsn()`.
Part of #10109
NO_DOC=bug fix
NO_CHANGELOG=next commit