nest() and unnest() support nested tables: nest() collapses each group
into a data frame stored in a list-column, and unnest() expands list-columns
back into rows.bt_default_threads() no longer calls parallel::detectCores() on every
verb (about 4 ms each); the detected core count is cached for the session.
Indexed subsets no longer call INTEGER() once per row while converting the
row index, which makes a 1e6-row indexed subset about 7x faster.
Grouping a single integer or logical key uses a dense table instead of the generic byte-key hash map, which is about 18x faster on 1e6 rows.
semimerge(), antimerge(), matchedkeys(), unmatchedkeys(),
intersectrows() and diffrows() now match factor keys by label.
Previously two factor keys were compared by their integer codes, so
factors with different levels silently returned the wrong rows.
merge() keeps a factor key a factor with the levels of x, plus any new
levels of y for right and full joins, as base::merge() does. It was
returned as character.
rbindfill() keeps a column that is a factor in every input holding it as
a factor, with the union of levels in input order, as base::rbind()
does. Factors were returned as character, including when a table without
the column was filled with NA.
aggregate() now recognises the native reducers (sum, mean, min,
max, var, sd, length) by the function itself rather than by how
it is spelled, so f <- min; aggregate(..., fun = f) gives the same
result as fun = min. Before, the spelling decided the code path, so an
all-NA group under na.rm = TRUE gave NA or Inf depending on the
call, and a user function named min was replaced by the native one.
aggregate() with a non-native fun keeps the value columns when the
input has no rows.
intersectrows() and diffrows() without by now compare whole rows on
every column of x. They called base::intersect()/base::setdiff(),
which treat a data frame as a list of columns, so they returned an empty
or malformed table.
towide() with the default fun = NULL now places each cell's value
instead of counting it, so towide(tolong(x, ...)) round-trips. Cells
holding several values are still counted with length(), now with a
message. Its argument documentation is rewritten.
cleannames() and repairnames() now always return syntactic column
names. Names starting with a digit get an x prefix ("2nd_col" becomes
"x2nd_col") and reserved words are made valid, so df$x2nd_col works
(#126).
split() on a non-data-frame data argument (the classic
split(vector, factor) base R idiom) now dispatches to base::split()
instead of erroring. Previously split() unconditionally treated its
first argument as a table, so attaching basetable broke
split(vector, factor) anywhere in a session, including inside
R Markdown/Quarto's own rendering pipeline, which calls split()
unqualified internally.
compact() drops NULL elements from a list, keeping other falsy values
(NA, 0, "") intact. The natural companion to map()/foldr() when
some calls legitimately return nothing (e.g. an empty API response) and the
NULLs need dropping before folding or row-binding the rest.count(sort = TRUE) now breaks ties (groups with equal counts) by
ascending group key instead of leaving them in the grouping engine's
internal enumeration order. That order was never part of the documented
contract and could silently change between basetable versions (it did,
between 1.3.1 and 1.4.0, as part of the grouping-key codec work), which
made count(sort = TRUE) output non-reproducible across versions for any
workflow that samples from, or otherwise depends on, row order among tied
counts.benchmarking vignette no longer hard-requires memory profiling: it
now checks capabilities("profmem") and falls back to timing-only
bench::mark() output (dropping the memory column/plot) when
Rprofmem() is unavailable, which was failing R CMD check on some
r-devel build machines.as_basetable() coerces a data frame, a list of equal-length columns, or a
matrix to the basetable class, and is_basetable() tests for it. Both are
now exported, so other packages can build directly on the basetable class
instead of relying on a verb to stamp it or reaching into internals.transliterate() help page no longer embeds literal Greek and Cyrillic
characters, which broke the PDF reference manual on CRAN's check machines.
The example now builds its sample strings with intToUtf8().stringi is no longer imported. removeaccents() and transliterate()
were its only users; they now fold text with tables extracted verbatim
from ICU's Latin-ASCII and Any-Latin; Latin-ASCII transliterators
(bench/make-translit-maps.R), so Latin output is unchanged and Greek and
Cyrillic match ICU except that an upper-case letter expanding to a digraph
stays upper case. basetable now imports only base and recommended
packages.merge() and the inner/left cases behind it) now probe the
build side in parallel and gather the atomic output columns on worker
threads. On the 1e6-row reference join this moves merge() from roughly
1.7x of data.table to about parity.merge(), semimerge(), antimerge() and updatemerge() on
such keys.merge() and
subset() on character data.orderrows() then counts and buckets the levels directly
rather than ranking every string.pick() and drop() select columns by shallow reference rather than
routing through the row materialiser, so narrowing a wide table no longer
copies the retained column data.build_frame, behind every sort, filter and the join
fallback) now gathers each character column's source pointers on worker
threads alongside the numeric columns, leaving only the SET_STRING_ELT
walk on the R thread. On the 1e6-row orderrows(by = c(<chr>, <num>)) this
roughly halves the materialisation step.subset() filter compacts its match list branch-free (write the
index, advance the cursor by the 0/1 predicate) instead of a mispredicting
if/push_back, bringing single-threaded subset(d, x > c) level with
data.table.casewhen(conditions, default = NA): vectorised multi-branch selection.
conditions is a named list of equal-length logical vectors, the same
shape as collapsevalues() -- the name of the first element that is TRUE
at a position is the value taken there; positions matching nothing get
default. Composes inside transform().subset() gains a by argument: the predicate is then evaluated within
each group, so aggregate references in it (mean(x), max(x), ...) are
per group. Kept rows are returned in their original order. This is
basetable's explicit answer to a grouped filter -- there is still no
stateful group_by().samplerows() and samplefrac() gain a by argument: sample n rows
(capped at the group size) or a fraction of each group. Sampled rows are
returned in their original order.basetable is now a self-contained data-manipulation package with a base-R interface and a bundled C++ execution engine. It has no external computation dependency. This is the first CRAN release.
data.table is no longer a dependency. Every verb runs on basetable's
own compiled engine. data.table and dplyr remain in
Suggests only, as a comparison target in the benchmark vignette.basetable rather than a data.table. A basetable is
an ordinary data frame with one extra class so it prints compactly and [
keeps the class and defaults to drop = FALSE. as.data.frame() returns a
plain frame. The as = argument of btread() and the file readers now
takes "basetable" in place of "data.table".bt_ prefix. aggregate(),
count(), uniquerows() and freq() now accept a single file path as
their first argument and route to the one-pass scan, so there is one name
per operation whether the input is a data frame or a delimited file.
bt_aggregate(), bt_count(), bt_distinct() and bt_freq() are removed.contains() / matches() (tidyselect selection helpers) are now
containstext() / matchestext(); glimpse() (tibble) is preview();
cummean() (dplyr) is cumavg(). distinct() is dropped -- uniquerows()
is the data-frame form and also takes a file path. The only names shared
with dplyr are now count() and pick(), both read as base-style verbs..Call kernels:
projection, row subsetting, ordering, distinct and duplicate detection,
grouping, grouped reducers (sum/mean/min/max/var/sd/n), all
join kinds (equi, semi, anti, update, cross, non-equi, range, rolling),
row-bind, and subset() predicate evaluation. Custom aggregation
functions still run as ordinary R closures per group.orderrows() sorts with a stable multi-column radix over order-preserving
integer codes (character columns ranked in strcmp order) instead of a
comparison sort; ordering is unchanged (C-locale byte order).subset() compiles a supported predicate (column and scalar references;
+ - * / ^ %%; comparisons; & | !; unary -; ifelse()) to a small
stack machine, with an eval() fallback for anything outside that grammar.setthreads():
grouped aggregation, the sort pipeline, the filter mask compaction and
column gather, and the join membership probe.btread() / btwrite(), memory-mapped,
RFC 4180, threaded parsing, optional ALTREP lazy columns), and
aggregate() / count() / uniquerows() / freq() on a file path fuse the
parse with the grouping so unused columns are never materialised.Measured against data.table at 1e6 rows on the reference machine (Linux, 8
threads); see the Benchmarks vignette for the full report.
data.table: uniquerows(), sd / var by
group at every cardinality, count and mean by group at high
cardinality, semimerge() (parity at low cardinality down to ~0.25x at
high), pick() / column projection, and subset() on the common
col <op> scalar shapes (single or &-joined comparisons over a numeric
column), now evaluated and materialised in one threaded pass with no
intermediate logical vector.merge(), mean by group at low cardinality,
rbindfill().data.table: string and integer orderrows() (~2x; the
radix byte passes are memory-bandwidth bound). data.table's parallel
radix sort is the one operation basetable does not yet match.aggregate() or count() allocates near zero where
the other engines allocate tens of megabytes.uniquerows() also takes a single file path: a fused one-pass scan that
returns the distinct combinations of cols without materialising the file.rbindfill() bulk-copies columns that already share a type across every
input and now keeps a shared column class such as Date.stack() is exported and consistently returns a basetable, masking
utils::stack() like the rest of the base-flavored API.inst/benchmarks/benchmark-scale.R: a size x cardinality harness against
base R, data.table and dplyr.btread(lazy = TRUE) now returns character columns as ALTREP too (not
only integer/double), so a lazy read parses nothing until a column is
touched. Combined with the parallel row indexer this makes "open a file"
and "read a few columns of a wide file" faster than both data.table::fread
and vroom (vroom still wins nothing here; fread keeps the lead only
when every value is parsed with all threads). See bench/RESULTS.md.btread() and btwrite(): a from-scratch delimited-file reader and
writer implemented in C++ (src/). The reader memory-maps the file, scans
it once for row boundaries with RFC 4180 quoting, guesses column types from
a bounded sample, and materialises typed vectors with numeric columns
filled by a std::thread pool. btread(lazy = TRUE) returns integer and
double columns as ALTREP vectors that parse their column only on first
access, which pairs well with col_select for wide files. btwrite()
formats disjoint row ranges into private buffers in parallel. Supports
delimiter sniffing, col_types, col_select, na, comment, skip,
n_max, trim_ws, and .gz input. See bench/benchmark-io.R.subset(data, select = -colname), negative bare-symbol column selection,
a base R subset() idiom, errored with "object 'colname' not found"
instead of dropping the column. select was being force-evaluated as an
ordinary argument (via is.null(select)/is.character(select)) before
its NSE form ever got a chance to run, and the column-index mask used to
evaluate NSE selections was unnamed. Rewritten to defer evaluation and
build a proper named index mask, matching select/subset and
select = -c(a, b) style multi-column negative selection too.reorder(). It was an exact duplicate of orderrows() (added in
0.6.0 to replace the dplyr-named arrange()), and it masked
stats::reorder() the same way the removed dplyr verbs masked dplyr's,
which silently broke ggplot2::aes(reorder(...)) for anyone with
basetable attached. Use orderrows() instead.split()'s keep.by now defaults to TRUE (was FALSE), so the
grouping column is kept in each piece by default, matching base R's
split() behavior on data frames. Pass keep.by = FALSE for the old
default.aggregate() no longer forces na.rm onto every fun. A custom
summary function with no na.rm argument (and no ...) previously
errored with "unused argument"; na.rm is now only forwarded when
fun actually accepts it.transform() gained a by argument. When supplied, each expression in
... is evaluated within every group instead of across the whole table
(e.g. transform(data, dev = x - mean(x), by = "g") computes the
deviation from each group's own mean), so the result still has one row
per input row. Use summaries() when you want one row per group instead.rbindfill()'s typeconflict argument now does something: "error"
(the default) checks every shared column across inputs up front and stops
with a message naming the column and the conflicting types before any
coercion happens; "coerce" skips the check and falls back to
data.table::rbindlist()'s usual coercion.R/verbs.R: filter(),
select(), rename(), arrange(), mutate(), transmute(),
summarise()/summarize(), distinct(), slice(), relocate(),
bind_rows(), and bind_cols(). These duplicated dplyr's verb names
exactly, which meant attaching basetable and dplyr in the same session
could silently mask either package's verb grammar depending on load
order. The underlying functionality already existed under base-flavored
names and continues to be exported: use subset(), pick(), reorder(),
transform() (with .keep = FALSE in place of transmute()),
summaries(), move(), and rbindfill() instead. bind_cols() had no
package-specific behavior beyond base R's cbind(); use cbind()
directly. Two functions had no existing base-flavored equivalent and are
now available under new names: renamecols() (was rename()) and
uniquerows() (was distinct()).removeaccents() and transliterate() deterministic across Linux,
Windows, and macOS by using ICU transliteration instead of the
platform-dependent iconv(..., "ASCII//TRANSLIT") behavior.R CMD build's temporary library on every
supported platform.transform(), mutate(), and transmute() now resolve ordinary values
from their true calling function, including when used inside a helper.value; the evaluated expression result is now assigned through a
standard-evaluation data.table setter.filter(), rename(),
distinct(), slice(), relocate(), and bind_rows().filter() to one logical mask in the common one-expression case
and avoided allocating an NA mask when no missing values are present.orderrows() and every
helper that uses the shared base ordering path.orderrows() through the package's data.table-backed ordering
implementation, removing an avoidable data.frame conversion while
preserving input immutability.Author:/Maintainer: fields from DESCRIPTION
(they had drifted out of sync with Authors@R, tripping an R CMD check --as-cran NOTE); both are now derived from Authors@R as usual.R-CMD-check.yaml workflow (macOS/Windows/Ubuntu,
release/devel/oldrel-1) and matching README badges.R CMD check --as-cran is clean apart from the two expected first-submission
notes (new submission, unable to verify current time).Initial development cycle covering performance, correctness, documentation, and CRAN readiness.
rollingmerge() reimplemented a rolling/nearest join with a hand-rolled
per-group R loop that measured at roughly 1500x the cost of data.table's
native roll= join; it now runs at parity.count(), duplicated_keys(), freq(), filldown(), fillup(),
split(), and summarise()/summarize()/summaries() were rewritten to
route through data.table's native grouping instead of base-R
interaction()/table()/split() idioms.inst/benchmarks/benchmark-all.R as a standing benchmark harness
tracking basetable against raw data.table.floordate()/
ceilingdate()/rounddate() were no-ops; denserank() ranked by
first-appearance order instead of sorted value; padleft()/padright()
ignored their pad argument; datediff() ignored units; dateseq()
errored whenever length.out was supplied; parsedatetime() dropped its
requested time zone; expandlevels() prepended instead of appended new
levels; emptycols() crashed on more than one row; towide()'s reshape
formula was backwards; nonequimerge()/rangemerge() silently performed a
plain equi-merge despite their names; removeduplicates(keep = "none")
deleted every row instead of keeping non-duplicates; equalrows(by =)
compared only the key columns; changedrows() had no actual
change-detection; changedcols() accepted a by argument it never used;
missingness() used a narrower "missing" definition than its sibling
functions; antimerge()/semimerge()/nonequimerge() did not validate an
empty by, unlike the rest of the merge family.distinct(), denserank(), completegrid().vignettes/functions-reference.qmd, a Quarto-authored PDF reference
vignette walking through every exported function with a real, runnable
example, organized into 14 progressive sections.R CMD check --as-cran passes with 0 errors/warnings (only the expected
"New submission" NOTE).cran-comments.md, CONTRIBUTING.md, URL/BugReports fields.