Papers on time-series anomaly detection report high F1 scores. The TSB-AD paper (Liu and Paparrizos, NeurIPS 2024, Datasets and Benchmarks track) says the field has ignored “flawed datasets, biased evaluation measures, and inconsistent benchmarking practices.” Its answer is a benchmark of 1,070 curated time series from 40 datasets, 40 algorithms, over 450 tuned variants and 10 evaluation measures. This post is a reading note on the paper, its public repo and the 2026 leaderboard. Every number comes from those sources. Result numbers are the authors’ own benchmark unless the text says otherwise.
Context: why the old numbers were suspect
Two earlier papers set up the problem. Wu and Keogh (arXiv:2009.13807) studied benchmarks “created by Yahoo, Numenta, NASA, etc.” and found that most of their exemplars “suffer from one or more of four flaws”. Their conclusion: “much of the apparent progress in recent years may be illusionary.” TSB-AD names the four flaws as triviality, anomaly density, mislabeling and run-to-failure bias.
Kim et al. (arXiv:2109.05257) went after the metric. They showed that point adjustment “has a great possibility of overestimating the detection performance”. Even a random anomaly score can pass for a top method under it.
TSB-AD works on both the data and the metric. It starts from 13 univariate and 20 multivariate public datasets and curates them down to 40 datasets. That gives 870 univariate series (TSB-AD-U) and 200 multivariate series (TSB-AD-M). The flaws it removes or fixes include mislabeling, run-to-failure bias (anomalies mostly at the end of a series, as in Yahoo), single-anomaly bias (as in UCR) and unrealistic anomaly ratios. Each construction step uses the consensus of four human annotators. An “algorithm test” then compares the scores of several detectors with the labels.
For scale, the Wu and Keogh archive has 250 curated series. TSB-AD has 1,070 (Table 1 of the paper). Fifteen percent of each dataset forms a tuning split and evaluation runs on the rest. Series are z-normalized unless the original implementation uses another normalization.
The multivariate part holds a few rigs a test engineer will recognise. GHL covers 3 reservoirs with temperature and level, and its anomalies are changes in max temperature or pump frequency. Genesis is a portable pick-and-place demonstrator. SWaT has 51 sensors and actuators, and its anomalies are behaviours under attack scenarios.
The metric: what point adjustment does
Point adjustment (PA) is a scoring rule. In the paper’s words, “detecting any point within an anomalous segment is considered as if all points within that segment were detected.” One lucky hit scores the whole segment.
The repo implements it in basic_metricor._adjust_predicts, in TSB_AD/evaluation/basic_metrics.py. This is the adjustment loop, quoted unmodified from commit 303dffb:
# TSB_AD/evaluation/basic_metrics.py (TheDatumOrg/TSB-AD@303dffb, lines 115-132)
actual = label > 0.1
anomaly_state = False
anomaly_count = 0
for i in range(len(score)):
if actual[i] and predict[i] and not anomaly_state:
anomaly_state = True
anomaly_count += 1
for j in range(i, 0, -1):
if not actual[j]:
break
else:
if not predict[j]:
predict[j] = True
latency += 1
elif not actual[i]:
anomaly_state = False
if anomaly_state:
predict[i] = True
actual marks the labeled anomalous points. When a predicted point lands inside a labeled segment and no segment is active, the loop sets anomaly_state and walks backwards, marking every earlier point of that segment as predicted. Then if anomaly_state: predict[i] = True marks each later point as predicted until a normal point ends the segment. The labels rewrite the predictions before F1 is computed.
The paper says random predictions can score close to one on PA-F1 as the average anomaly length grows. It tests this with Random (C), a continuous uniform random score between 0 and 1. Compared with the scores of 32 detectors, “the random score achieves a ranking of 26 under the PA-F1 measure.” The text does not say whether that is 26th of 32 or of 33, so read it as written.
Table 5 puts the same methods under PA-F1 and under VUS-PR on TSB-AD-U:
- SR: 0.87 on PA-F1, 0.32 on VUS-PR. It has the highest PA-F1 in the table.
- TimesFM: 0.84 and 0.30. Chronos: 0.83 and 0.27.
- CNN: 0.78 and 0.34.
- Sub-PCA: 0.56 and 0.42.
- AnomalyTransformer: 0.53 and 0.12.
The appendix adds that neural methods generally score higher after point adjustment.
Labels are the other half. Earlier benchmarks took an AUC-ROC above 0.8 from at least one detector as a sign of good labels. The paper shows a case with an AUC-ROC of 0.97 that still has two false negatives.
TSB-AD scores with VUS-PR. VUS-ROC and VUS-PR add a tolerance buffer around anomaly boundaries and use continuous values instead of binary labels. After comparing the 10 measures, the paper judges VUS-PR best on three counts: least sensitive to lags, unbiased across scenarios, and consistent under similar cases. Thresholds are out of scope. The paper calls them “an orthogonal problem”.
Results: what won on TSB-AD
All scores in this section are mean VUS-PR from the authors’ benchmark.
On the univariate set, Sub-PCA leads with 0.42. KShapeAD follows at 0.40, then POLY, Series2Graph and MOMENT (FT) at 0.39. The bottom of the table is LOF at 0.17 and AnomalyTransformer at 0.12. In the top 12, the paper counts more than half as statistical methods, with two neural methods (USAD, CNN) and one foundation model (MOMENT).
Sub-PCA is a small piece of code. This is its wrapper in TSB_AD/model_wrapper.py:
# TSB_AD/model_wrapper.py (TheDatumOrg/TSB-AD@303dffb, lines 134-140)
def run_Sub_PCA(data, periodicity=1, n_components=None, n_jobs=1):
from .models.PCA import PCA
slidingWindow = find_length_rank(data, rank=periodicity)
clf = PCA(slidingWindow = slidingWindow, n_components=n_components)
clf.fit(data)
score = clf.decision_scores_
return score.ravel()
The window length comes from the autocorrelation function. The 1st, 2nd and 3rd strongest periods are the three candidates, and find_length_rank reads at most the first 20,000 points. Each window becomes one point of a pseudo-multivariate series, and PCA runs on that. The PCA detector is adapted from pyod. The tuned setting in HP_list.py is periodicity: 1 and n_components: None.
On the multivariate set, four methods tie at 0.31: CNN, OmniAnomaly, PCA and LSTMAD. The paper’s reading is that “simpler architectures such as CNNs and LSTMs generally outperform more complex designs, such as advanced transformer architectures.” AnomalyTransformer scores 0.12 there too.
The winner depends on the anomaly shape. For point anomalies, TimesFM and Chronos rank first and third. For sequence anomalies, statistical methods dominate, with Sub-KNN and POLY ahead. The paper’s explanation is that a forecaster “estimates only one new value per step using a limited look-back window”. These models forecast from a sliding context and use the squared error between prediction and actual value as the anomaly score.
No single model wins everywhere. Ranked by mean VUS-PR, Sub-PCA is first. Ranked by average rank, the paper puts Sub-POLY first. Some methods “completely fail” on certain datasets while excelling on others.
Contamination is the check to remember. The authors compared MOMENT’s score on TSB-AD-U with its score on MOMENT’s own held-out evaluation subset. MOMENT (FT) went from 0.39 to 0.14 and MOMENT (ZS) from 0.38 to 0.12. KShapeAD went from 0.40 to 0.32 on the same comparison. Only MOMENT was checked this way. The paper gives no such numbers for TimesFM, Chronos or Lag-Llama.
Runtimes appear only in a bar chart, so this post quotes none. In words: statistical methods are generally fastest, foundation models slowest, and CNN and LSTM are effective and fast.
Since the paper: the 2026 leaderboard
The repo README says the leaderboard has accepted community submissions since Apr. 1, 2026. The project page shows “Last updated: October 20, 2026”. Everything in this section is a leaderboard entry, not a result of the peer-reviewed paper. The sources do not say how entries are verified.
The leaderboard ranks by average VUS-PR. These are the top entries added after the paper, scored on TSB-AD-U and TSB-AD-M:
- CHARM (kNN): 0.68 and 0.54, pretrained.
- Syntropic: 0.67 and no entry, pretrained.
- CHARM (ZS): 0.61 and 0.46, pretrained.
- VERITY-AD: 0.61 and no entry, not pretrained.
The same table lists a “PCA” row at 0.42 and 0.31, and AnomalyTransformer at 0.12 and 0.12. The PCA row matches Sub-PCA on the univariate set and PCA on the multivariate set. The page does not say whether it merges the two.
Two caveats. The Time-RCD+MAFT and VERITY-AD entries link to compiled .so files in the repo, not Python source, so their method cannot be read there. And contamination was only checked for MOMENT in the paper, so the pretrained entries carry the same open question.
“Simple beats deep” was true in the 2024 paper. The 2026 leaderboard now has pretrained entries above the 2024 Sub-PCA row on the univariate set. Both statements are about different submissions on different dates.
Lessons and wrap-up
- Ask which metric produced an F1 before comparing detectors. Under PA-F1, a uniform random score ranked 26 against 32 detectors.
- Start with a statistical baseline such as subsequence PCA. It led TSB-AD-U at 0.42 VUS-PR while AnomalyTransformer scored 0.12.
- Match the detector to the anomaly shape. Forecasting foundation models did best on point anomalies and worse on long sequence anomalies.
- Treat zero-shot scores from pretrained models as suspect until they are tested on data the model cannot have seen. MOMENT fell from 0.39 to 0.14 on its own held-out subset.
- Labels decide the result. TSB-AD used four annotators and an algorithm test at each step, and its authors still call their manual inspection a limitation.
To try it, pip install TSB-AD installs the package. The README evaluates a detector in 10 lines: run_Unsupervise_AD(AD_Name, data) produces the score and get_metrics(output, label) returns nine measures. PATE is not among them. The example runner thresholds at the mean plus 3 standard deviations of the min-max scaled score. The sources do not call that threshold a recommendation.
The setting on this site’s side is nailer fixture life tests. Their data lives in InfluxDB and ClickHouse, and a pipeline collects tool and fixture signals for every cycle, such as current, voltage, pressure and temperature. Samples come from LabVIEW over MQTT, at a lower rate while the tool is idle and a higher one during a firing. The pipeline groups samples by cycle, splits a cycle into fire attempts, and judges each attempt against a rolling baseline of recent healthy firings from the same fixture and tool. Each attempt is recorded as a healthy fire, a weak fire, a misfire with no current, or noise. Raw samples and per-cycle summaries are stored side by side. The repo behind it is private, so this post describes the flow and nothing more.
Those verdicts come from rules, not from a labeled anomaly set scored with VUS-PR. Whether TSB-AD detectors and the VUS-PR measure fit this data is an open question, and this post reports no run of either on it. If you run detectors on test-rig data, check two things first: how the anomalies were labeled, and which metric the F1 comes from.