001 Notes

ORB Python-To-Rust Speedup Record

Scope And Limitations

This note records the available benchmark evidence for a Python-to-Rust ORB backtester migration. It keeps the useful timing data, parity checks, design facts, accepted and rejected optimization history, and claim boundaries in one public article.

Important limitations:

The old Python ORB implementation was not available in the reviewed evidence.
A direct old-Python-ORB vs current-Rust-ORB end-to-end benchmark was not available.
The strongest 100x-class evidence is stage-level, not full ORB end-to-end.
This is an audit of available benchmark artifacts, not a complete commit-by-commit migration history.

Executive Summary

The Rust backtester became much faster because the implementation stopped doing expensive Python-style repeated work:

Load market data into typed arrays once.
Cache loaded OHLCV data in a binary format.
Build higher timeframes once, session by session.
Prepare one ORB entry per candidate/session.
Evaluate many stop/RR combinations in one future-bar scan.
Track unresolved combinations with u64 bitmasks.
Use AVX2 threshold comparison where available.
Reuse worker threads through persistent pools.
Profile scheduler choices and save runtime baselines.
Materialize full trade rows only for selected rows or inspection.

The measured stage-level Rust wins available in the record:

Base data loading              25.192x | #####
Higher-timeframe resample      33.297x | #######
Range discovery                92.306x | ##################
Chunk analytics build           2.128x | #
Dense prepared reconstruction   68.817x | ##############
Finalize event batch build     152.802x | ###############################

Scale: one # is approximately 5x speedup.

The two strongest accepted stage wins are:

range discovery: 118.732s Python -> 1.286s Rust, 92.306x.
finalize event batch build: 21.037s Python -> 0.138s Rust, 152.802x.

The speedup history has three layers in this document:

Within Python: caching, batching, pruning, dense reducer work, and finalize/OOS cleanup.
Python to Rust: six accepted stage ports with measured Python-vs-Rust timings.
Within Rust: typed arrays, binary cache, prepared entries, combo batching, pending masks, AVX2, persistent pools, and scheduler tuning.

Those are the concrete measurements behind the “100x-class” statement. The document does not claim that every full ORB run is always 100x faster end to end.

Evidence Inventory

Evidence used:

Current Rust backtester implementation and ORB profile artifacts.
Current ORB family documentation for single-timeframe, multi-timeframe, human-readable, and multi-timeframe human-readable variants.
Historical benchmark records for the within_session_day_high family.
Saved optimization report snapshot from 2026-05-01.

Snapshot metadata:

Field	Value
family	`within_session_day_high`
report date	`2026-05-01`
rustc version	`1.95.0`
cargo version	`1.95.0`
accepted Python/runtime optimizations	`12`
accepted Rust stage ports	`6`
rejected DuckDB/Rust experiments	`7`

Evidence quality key:

Label	Meaning
Direct measured	A saved benchmark gives Python time, Rust time, speedup, and parity status.
Current ORB profile	A saved ORB profile gives Rust timings and scheduler tuning, but not Python comparison.
Implementation verified	The current Rust implementation contains the named structure, function, or algorithm.
Inferred	Derived from implementation and known Python bottleneck shape; old Python implementation was not available.
Missing	Needed for complete historical proof but not available in the reviewed evidence.

Visual Architecture Change

flowchart LR
    A["Old inferred Python-style path"] --> B["Load DB rows and Python objects"]
    B --> C["Rebuild or slice timeframes repeatedly"]
    C --> D["Discover signal/range repeatedly"]
    D --> E["Scan future bars per stop/RR row"]
    E --> F["Materialize rows early"]
    F --> G["Aggregate with dict/list/DataFrame overhead"]

    H["Current Rust path"] --> I["Load typed OHLCV arrays or binary cache"]
    I --> J["Build resident session-aware frames once"]
    J --> K["Prepare entry per candidate/session"]
    K --> L["Flatten SL/RR ComboSpec grid"]
    L --> M["One batched future scan"]
    M --> N["u64 masks, AVX2 compares, early exit"]
    N --> O["Summaries first, trades only for selected rows"]

Current Rust ORB Pipeline

flowchart TD
    DB["SQLite DB"] --> RO["Read-only loader"]
    RO --> BC["Binary symbol cache"]
    BC --> OHLCV["OhlcvFrame typed arrays"]
    OHLCV --> RES["Session-aware resample cache"]
    RES --> RSI["RSI planes, if requested"]
    RSI --> BUNDLE["ResidentBundle"]
    BUNDLE --> PREP["prepare_entries"]
    PREP --> ENTRY["PreparedEntry per session"]
    ENTRY --> GRID["Flattened ComboSpec SL/RR grid"]
    GRID --> SCRATCH["WorkerScratch arrays"]
    SCRATCH --> MASK["Pending masks"]
    MASK --> SIMD["AVX2/scalar compare mask"]
    SIMD --> ACC["SummaryAccumulator"]
    ACC --> LB["Leaderboard rows"]
    LB --> TRADES["Selected trade rows"]

Optimization Count Chart

Accepted Python/runtime optimizations   12 | ############
Accepted Rust stage ports                6 | ######
Rejected DuckDB/Rust experiments         7 | #######

Interpretation:

There was a Python optimization phase before the Rust stage-port phase.
Not every Rust or DuckDB experiment was accepted.
The final approach kept only native slices that were fast enough and parity-safe enough.

Speedup Layer 1: Within Python Changes

This phase made the existing Python pipeline faster before the Rust replacement work. The important point is that the later Rust numbers were not measured against a completely unoptimized Python baseline. Python had already been improved through caching, batching, pruning, and less repeated object work.

Python Phase Visual

Timeframe-loading slice
Before Python cleanup     44.595s | ######################
After Python cleanup      20.849s | ##########
Speedup: 2.139x

RR/stop grid slice
Before Python batching     2.268s | ###########
After Python batching      0.950s | #####
Speedup: 2.388x

Finalize/OOS path
Early finalize state    1370.690s | #######################################################
After major cleanup      519.537s | #####################
Derived speedup: 2.638x

Later finalize cleanup chain
Before smaller cleanups   494.746s | ####################
After smaller cleanups    351.280s | ##############
Derived speedup: 1.408x

Scale: timeframe-loading uses approximately 2 seconds per #; RR/stop batching uses approximately 0.2 seconds per #; finalize uses approximately 25 seconds per #. The finalize scale is compressed so the chart stays readable.

What Changed Inside Python

#	Python-side change	What changed operationally	Why it was faster	Measurement / status
1	`1m` base-loading contract plus in-memory resample	The pipeline standardized on loading canonical `1m` bars and deriving higher timeframes in memory.	Avoided repeatedly loading or rebuilding equivalent timeframe data through slower row/object paths.	`44.595s -> 20.849s`, `2.139x`, kept
2	RR and stop grid batching	Stop-loss and risk/reward variants were grouped instead of being handled as fully separate small jobs.	Reduced repeated loop dispatch, repeated future scans, and repeated Python object setup around the same market slice.	`2.268s -> 0.950s`, `2.388x`, kept
3	Coordinator/finalize checkpoint plus OOS cleanup	Finalize work was made more checkpointable and the out-of-sample cleanup path was reduced.	Avoided redoing already-completed chunks and reduced repeated post-processing around validation output.	`1370.690s -> 519.537s`, kept
4	Finalize input cache plus setup signal cache	Stable finalize inputs and setup/signal work were cached between repeated runs.	Saved repeated preparation of the same signal and validation inputs.	`494.746s -> 483.357s`, kept
5	Dense reducer merge redesign	Reducer merging was reshaped toward dense intermediate structures.	Reduced overhead from nested Python containers during merge-heavy stages.	`483.357s -> 392.279s`, kept
6	Prepared-outcome / OOS formatting cleanup	Prepared outcome and OOS formatting work was trimmed.	Pushed less formatting through hot paths and reduced conversion work before final output.	`392.279s -> 366.200s`, kept
7	Finalize input cache v2	The finalize input cache was refined.	Increased reuse of stable inputs and reduced remaining cache miss/rebuild work.	`366.660s -> 362.340s`, kept
8	OOS probability map cleanup	OOS probability-map handling was simplified.	Reduced lookup/formatting overhead around validation probability data.	`362.340s -> 360.260s`, kept
9	Dense-arrays to prepared cleanup	Dense-array to prepared-output conversion was cleaned up.	Reduced conversion overhead when moving from numeric arrays back into prepared Python output objects.	`360.260s -> 358.400s`, kept
10	OOS rank tail-prune	Lower-value tail candidates were pruned before later ranking/finalize work.	Prevented the slowest downstream code from processing candidates that could not matter to the final ranking.	`358.400s -> 352.780s`, kept
11	Split finalize cache	The finalize cache was split by more stable sub-inputs.	Allowed reuse of sub-results when only part of the finalize input changed.	`352.780s -> 351.280s`, kept
12	Worker batch shaping	Worker batch shape was tuned.	Improved the target hotspot, but the saved report says end-to-end runtime stayed flat.	kept as a hotspot improvement, no material e2e gain

Python Lessons That Carried Into Rust

Loading and resampling needed to be resident and reused.
Stop/RR grids needed to be batched, not expanded into one independent row at a time.
Finalize and OOS work had to avoid repeated setup and repeated formatting.
Dense numeric structures were better than nested Python dictionaries and lists.
Pruning before expensive ranking/finalize work mattered.
Worker shaping mattered, but Python still paid high process, serialization, and object overhead.

Python Phase Takeaway

The Python phase removed obvious waste and delivered real wins, but it did not remove Python’s core bottleneck model: object-heavy row materialization, Python-level loops, nested containers, repeated conversion, process/serialization overhead, and expensive final formatting. That is why the Rust phase still had room for 25x to 152x stage wins.

Accepted Python And Runtime Optimizations

These were applied before or alongside the Rust-stage work. They matter because the Rust speedups were measured against an already-improved Python path, not necessarily the slowest original implementation.

#	Optimization	Result	Kept
1	`1m` base-loading contract plus in-memory resample	`44.595s -> 20.849s`, `2.139x`	yes
2	RR and stop grid batching	`2.268s -> 0.950s`, `2.388x`	yes
3	coordinator/finalize checkpoint plus OOS cleanup	finalize `1370.690s -> 519.537s`	yes
4	finalize input cache plus setup signal cache	`494.746s -> 483.357s`	yes
5	dense reducer merge redesign	`483.357s -> 392.279s`	yes
6	prepared-outcome / OOS formatting cleanup	`392.279s -> 366.200s`	yes
7	finalize input cache v2	`366.660s -> 362.340s`	yes
8	OOS probability map cleanup	`362.340s -> 360.260s`	yes
9	dense-arrays to prepared cleanup	`360.260s -> 358.400s`	yes
10	OOS rank tail-prune	`358.400s -> 352.780s`	yes
11	split finalize cache	`352.780s -> 351.280s`	yes
12	worker batch shaping	target hotspot improved, end-to-end flat	no material e2e gain

Visual ladder:

Initial representative loading slice   44.595s
After 1m base/in-memory resample        20.849s

Finalize path early state             1370.690s
After checkpoint/OOS cleanup           519.537s
After dense reducer redesign           392.279s
After prepared/OOS cleanup             366.200s
After final smaller cleanups           351.280s

Accepted Rust Stage Ports

These are the six accepted Rust slices from the benchmark history.

#	Stage	Python	Rust	Speedup	Parity / decision
1	Base data loading	`36.899s`	`1.465s`	`25.192x`	accepted, exact summary/signature match
2	Higher-timeframe resample	`60.053s`	`1.804s`	`33.297x`	accepted, exact summary/signature match
3	Range discovery	`118.732s`	`1.286s`	`92.306x`	accepted, exact summary/signature match
4	Chunk analytics build	`17.204s`	`8.084s`	`2.128x`	accepted, exact `ChunkAnalyticsSummary` parity
5	Dense prepared reconstruction	`29.256s`	`0.425s`	`68.817x`	accepted as hotspot upper-bound evidence, row-level summary/signature match
6	Finalize event batch build	`21.037s`	`0.138s`	`152.802x`	accepted, main and supplement parity true

Python Time vs Rust Time Chart

Scale: one # is approximately 5 seconds. Tiny Rust bars below one unit are marked with ..

Base data loading
Python 36.899s | #######
Rust    1.465s | .

Higher-timeframe resample
Python 60.053s | ############
Rust    1.804s | .

Range discovery
Python 118.732s | ########################
Rust     1.286s | .

Chunk analytics build
Python 17.204s | ###
Rust    8.084s | ##

Dense prepared reconstruction
Python 29.256s | ######
Rust    0.425s | .

Finalize event batch build
Python 21.037s | ####
Rust    0.138s | .

Speedup Layer 2: Within Rust Changes

This section separates two Rust ideas:

Python-to-Rust stage ports: moving a whole hot slice from Python into Rust.
Within-Rust optimization: making the Rust implementation itself faster through layout, batching, SIMD, scheduler tuning, and cache reuse.

The accepted Rust stage-port table above proves the stage-level Python-to-Rust wins. The table below explains the Rust-internal design changes that made those ports fast enough to keep.

Rust Phase Visual

flowchart LR
    A["Naive native port risk"] --> B["Still repeats work"]
    B --> C["Still scans per combo"]
    C --> D["Still allocates per job"]
    D --> E["Still has scheduler imbalance"]

    F["Optimized Rust path"] --> G["Resident typed arrays"]
    G --> H["Prepared entries"]
    H --> I["Flattened combo blocks"]
    I --> J["Scratch reuse and pending masks"]
    J --> K["AVX2 compare masks"]
    K --> L["Persistent pools and tuned scheduler"]

Current ORB Rust Pass-To-Pass Tuning Evidence

The current ORB profile artifact shows that tuning inside Rust matters even after the port exists.

Metric	`pass0_baseline`	`pass1_reprofile`	Improvement
Scheduler	`session_major`	`hybrid_blocked`	changed work partitioning
Workers	`8`	`16`	more parallelism
Session block	`1`	`24`	larger session batches
Param block	`8`	`8`	unchanged
Load/prep	`3087us`	`1233us`	`2.504x` faster
Signal eval	`499us`	`472us`	`1.057x` faster
Combo eval	`2572us`	`760us`	`3.384x` faster
Total wall	`3073us`	`1233us`	`2.492x` faster

Visual:

Combo evaluation
pass0 baseline   2572us | ##########################
pass1 tuned       760us | ########

Total wall
pass0 baseline   3073us | ###############################
pass1 tuned      1233us | ############

Scale: one # is approximately 100us.

Important interpretation:

This is a current ORB profile comparison, not an old Python-vs-Rust comparison.
It proves that Rust still needed scheduler/block-size tuning after the core port.
pass2_reprofile is not shown as an improvement because it used a denser stress shape and produced 21964us combo evaluation. That pass is useful for hotspot discovery, not for claiming a speedup over pass1.

What Changed Inside Rust

#	Rust-side change	What changed operationally	Why it was faster	Evidence
1	Columnar `OhlcvFrame` layout	Market data is held as `Vec<u32>`, `Vec<u16>`, and `Vec<f32>` columns rather than row objects.	Hot loops walk compact contiguous arrays with better cache locality and less per-row overhead.	implementation verified; base loading `25.192x` and resample `33.297x` stage evidence
2	Binary OHLCV cache	Loaded symbols can be stored in `RBCACHE1` binary cache format with typed arrays and session ranges.	Later runs avoid repeated SQLite row conversion and rebuild typed vectors directly.	implementation verified; base loading stage evidence
3	Read-only SQLite loader settings	Loader uses `query_only`, memory temp store, larger cache size, and mmap size.	Optimizes the DB path for read-heavy backtesting instead of generic SQLite behavior.	implementation verified
4	Session-aware resampling	Resampling happens inside session boundaries with session ranges preserved.	Higher timeframes are built once and reused without crossing sessions or rebuilding per candidate.	implementation verified; resample `33.297x`
5	Resident frame and RSI caches	`ResidentBundle`, `ResidentFrame`, and `RsiPlane` keep timeframes/features resident.	Feature construction is reused across variants instead of recomputed through every candidate path.	implementation verified
6	Prepared ORB entries	`prepare_entries` builds one numeric entry record per candidate/session before combo evaluation.	Signal discovery is separated from stop/RR evaluation, avoiding candidate x combo repeated signal work.	implementation verified; ORB effect inferred
7	Flattened `ComboSpec` grids	Stop/RR combinations are flattened into blocks.	The evaluator can process many combinations together instead of dispatching one Python-like row at a time.	implementation verified
8	`WorkerScratch` reuse	Stop prices, target prices, risks, metrics, and masks reuse per-worker buffers.	Avoids repeated allocation and keeps combo arrays hot in memory.	implementation verified
9	`u64` pending masks	Up to 64 unresolved combinations are tracked in one word.	Resolved combos stop consuming checks, and early exit ends the future-bar scan when all combos resolve.	implementation verified
10	AVX2 compare masks	Threshold checks compare 8 `f32` values at a time when AVX2 is available.	Stop/target checks become vector operations that feed directly into the pending mask format.	implementation verified
11	Scalar fallback	Non-AVX2 machines use the same mask contract through scalar loops.	Keeps behavior portable without splitting the evaluation model.	implementation verified
12	Persistent worker pools	Worker threads live across jobs and receive work through channels.	Avoids repeated thread creation during benchmark/search workloads.	implementation verified
13	Scheduler modes	`session_major`, `param_major`, `hybrid_blocked`, and `thread_per_param` support different workload shapes.	Work can be split along the dominant dimension instead of forcing one scheduler on every grid.	implementation verified; ORB profile pass evidence
14	Cached selected tuning	Profile/baseline artifacts record selected scheduler and block settings.	Later runs can reuse known-good choices instead of rediscovering them from scratch.	current ORB profile evidence
15	Summary-first output	Rust accumulates compact summaries and materializes trade rows only for selected/inspection paths.	Avoids Python-style full row expansion across every intermediate candidate.	implementation verified; dense prepared and event batch stage evidence

Rust Phase Takeaway

The Rust speedup was not just “rewrite Python in Rust.” The fast path came from changing the execution model:

Row objects became typed arrays.
Repeated data loading became resident caches.
Repeated timeframe construction became session-aware frame reuse.
Repeated signal work became prepared entries.
Per-combo future scans became batched combo blocks.
Per-combo state became bitmasks.
Scalar threshold checks became AVX2 masks when available.
Per-job thread creation became persistent worker pools.
One scheduler became a tuned scheduler matrix.
Full row materialization moved to selected output paths.

That is the reason the accepted Rust stages reached 25.192x, 33.297x, 92.306x, 68.817x, and 152.802x, while rejected native attempts were excluded when they were slower or failed parity.

Benchmark Card 1: Base Data Loading

Stage: base_data_loading

Scope:

Symbols: BANKNIFTY, FINNIFTY, NIFTY, SENSEX
Timeframe: 1m
Iterations: 2

Result:

Python mean: 36.899s
Rust mean: 1.465s
Speedup: 25.192x
Exact summary/signature match: True

Per-symbol parity:

Symbol	Match	Bars	Sessions	Timeframe	Signature
BANKNIFTY	True	`462814`	`1240`	`1`	`14979702488594476994`
FINNIFTY	True	`462815`	`1240`	`1`	`12830231235636915535`
NIFTY	True	`462862`	`1240`	`1`	`16746711586037283007`
SENSEX	True	`462854`	`1241`	`1`	`7164122480827181626`

Why it matters for ORB:

ORB runs need the same raw intraday market data.
The Rust backtester keeps this in resident typed arrays.
Repeated broad searches avoid repeated Python/SQLite row conversion.

Benchmark Card 2: Higher-Timeframe Resample

Stage: resample_symbol_data

Scope:

Symbols: BANKNIFTY, FINNIFTY, NIFTY, SENSEX
Target timeframes: 2, 3, 5, 7, 10, 15
Iterations: 2

Result:

Python mean: 60.053s
Rust mean: 1.804s
Speedup: 33.297x
Exact summary/signature match: True

Compact parity matrix:

Symbol	TF	Match	Bars	Sessions	Signature
BANKNIFTY	2	True	`232021`	`1240`	`10623403372796047685`
BANKNIFTY	3	True	`154274`	`1240`	`11683861620331284709`
BANKNIFTY	5	True	`92565`	`1240`	`15572291281304323151`
BANKNIFTY	7	True	`66648`	`1240`	`13119536419317246893`
BANKNIFTY	10	True	`46899`	`1240`	`3451223347036751332`
BANKNIFTY	15	True	`30856`	`1240`	`10196389433094323293`
FINNIFTY	2	True	`232022`	`1240`	`6773245237344672967`
FINNIFTY	3	True	`154274`	`1240`	`5452828566907294839`
FINNIFTY	5	True	`92565`	`1240`	`14013925004150641618`
FINNIFTY	7	True	`66648`	`1240`	`6484163754068440467`
FINNIFTY	10	True	`46899`	`1240`	`5091172562065009978`
FINNIFTY	15	True	`30856`	`1240`	`8323565727100361650`
NIFTY	2	True	`232045`	`1240`	`6269823062222553913`
NIFTY	3	True	`154289`	`1240`	`91100024418122270`
NIFTY	5	True	`92575`	`1240`	`10165090913439292164`
NIFTY	7	True	`66655`	`1240`	`15451707410263674778`
NIFTY	10	True	`46904`	`1240`	`6016081311971001549`
NIFTY	15	True	`30859`	`1240`	`7726354906344598091`
SENSEX	2	True	`232041`	`1241`	`12073356468092182653`
SENSEX	3	True	`154288`	`1241`	`3249127358710964883`
SENSEX	5	True	`92574`	`1241`	`1796560672201851377`
SENSEX	7	True	`66654`	`1241`	`9945275399724493284`
SENSEX	10	True	`46904`	`1241`	`6752291735196555283`
SENSEX	15	True	`30859`	`1241`	`638337056868199258`

Why it matters for ORB:

ORB depends heavily on 1m, 2m, 3m, 5m, 10m, 15m, 30m, and other timeframes.
Multi-timeframe ORB is only practical if higher-timeframe data is not rebuilt per candidate.
The current Rust resampler keeps candles inside session boundaries and anchors from the session open.

Benchmark Card 3: Range Discovery

Stage: range discovery

Scope:

Symbol: BANKNIFTY
Study timeframe: 1m
Slice shape: 2 policies x (1 red-close mode + 4 low-break rules x windows 1..10) x 3 boundaries
Detector calls: 246
Iterations: 2

Result:

Python single pass: 117.49s
Rust single pass: 1.267s
Python mean: 118.732s
Rust mean: 1.286s
Speedup: 92.306x
Total benchmark wall clock: 240.037s
Exact summary/signature match: True
Accepted: True

Why it matters for ORB:

ORB opening-range and breakout discovery is range/event discovery work.
Moving this class of loop into typed Rust removes the Python row-iteration bottleneck.
This is one of the two strongest accepted 100x-class measurements.

Benchmark Card 4: Chunk Analytics Build

Stage: chunk analytics

Scope:

Range rows: 6293
Attempt rows: 145760
Transition rows: 13529
Timed iterations: 5

Result:

Warmup Python: 16.771s
Warmup Rust: 7.887s
Python mean: 17.204s
Rust mean: 8.084s
Speedup: 2.128x
Total wall clock: 162.444s
Exact ChunkAnalyticsSummary parity: True
Decision: accepted

Interpretation:

This stage improved, but it did not produce a 100x-class gain.
It is included to avoid cherry-picking only the biggest wins.
It shows the audit process kept smaller wins when parity was exact.

Benchmark Card 5: Dense Prepared Reconstruction

Stage: dense prepared reconstruction around _dense_arrays_to_prepared_outcomes

Important read:

Current production Python materialized full nested Python dict/list structures.
The Rust prototype computed exact row-level summary/signature over the same reducer payload.
Therefore 68.817x is a hotspot upper bound, not a guaranteed end-to-end drop-in speedup.
The apples-to-apples reducer-summary loop speedup was even higher: 310.187x.

Correctness:

Row-level summary/signature match: True

Timing:

Python production _dense_arrays_to_prepared_outcomes mean: 29.256403s
Python reducer-summary mean: 131.871543s
Rust reducer-summary mean: 0.425135s

Speedup:

Rust vs Python reducer-summary: 310.187x
Rust vs Python production prepared-path: 68.817x

Row counts included:

Row class	Rows
baseline	`1200`
baseline-by-year	`7200`
single	`77521`
single-by-year	`460965`
pair	`1700252`
pair-by-year	`9583646`
triple	`2492830`
triple-by-year	`12569630`

Why it matters for ORB:

ORB broad searches can generate large intermediate grids.
Avoiding Python dict/list reconstruction is essential once candidate counts grow.
The current Rust ORB design summarizes first and materializes selected trades later.

Benchmark Card 6: Finalize Event Batch Build

Stage: finalized event batch build

Scope:

Status: accepted
Selected chunk count: 60
Timed worker-side call count: 300
Supplement call count: 7
Trimmed to first 30 chunks: False
Selection reason: full_60_chunks python one-pass 20.956s
Iterations: 12

Result:

Python one-pass: 20.956s
Rust one-pass: 0.080s
Python mean: 21.037s
Rust mean: 0.138s
Speedup: 152.802x
Main parity: True
Supplement parity: True
Total benchmark wall clock: 274.066s

Why it matters for ORB:

This is the strongest accepted 100x-class stage result.
It validates the design principle: compact event batches beat Python object expansion.
ORB inspection and leaderboard workflows benefit from this same “defer heavy rows until needed” approach.

Rejected Or Conditional Experiments

This is important: the migration was not “Rust everything blindly.” Several native or DuckDB experiments were rejected or only conditionally useful.

Experiment	Python	Candidate	Outcome	Reason
DuckDB final consensus on large sweep	`1.558s`	`0.467s`	conditional keep	useful only on large consensus-shaped aggregation
DuckDB dense outcome aggregation	n/a	much slower	reject	`12x` to `15x` slower than Python reducer path
Dense reducer merge in Rust	`18.340s`	`27.568s`	reject	slower and failed `reducer_signature` parity
OOS candidate selection hybrid	`19.163s`	`48.880s`	reject	exact but Python-side flattening dominated
Grouped row scoring hybrid	`48.136s`	`48.461s`	reject	exact mode slightly slower than Python
Prediction-cell hybrid	`0.825s`	`1.011s`	reject	slower and not row-exact
Finalize strategy evaluation in Rust	`50.442s`	`0.921s`	reject	very fast but failed exact `EvalSummarySet` parity

Rejected details from benchmark reports:

Dense reducer merge:
- payload count: 60
- selected shard count: 60
- Python mean: 18.34s
- Rust mean: 27.568s
- speedup: 0.665x
- exact summary/signature match: False
- mismatch field: reducer_signature
- decision: rejected
OOS candidate benchmark:
- Python build_oos_validation mean: 19.162655s
- Rust-hybrid build_oos_validation mean: 48.880381s
- speedup: 0.392x
- exact row equality: True
- canonical match: True
- row count: 430
- rejected because integration overhead made it slower.
Finalized strategy evaluation:
- stage: _evaluate_local_variants evaluation core only
- chunk count used: 60
- group count used: 300
- summary record count: 2700
- repeat count used: 4
- Python mean seconds: 50.442
- Rust mean seconds: 0.921
- speedup: 54.792x
- parity exact: false
- accepted: false
- rejected because Rust EvalSummarySet parity did not match Python CPU reference.
Attempt analysis:
- symbol: BANKNIFTY
- timeframe: 5m
- confirmed input count: 4839
- Python mean: 5.052s
- Rust mean: 0.036s
- speedup: 139.656x
- exact summary/signature match: False
- not promoted as accepted evidence because parity failed.

Visual accepted/rejected contrast:

Accepted because fast and parity-safe:
base loading               25.192x  parity True
resample                   33.297x  parity True
range discovery            92.306x  parity True
chunk analytics             2.128x  parity True
dense prepared             68.817x  signature True, hotspot upper-bound
event batch build         152.802x  parity True

Rejected or conditional:
dense reducer merge         0.665x  parity False
OOS candidate hybrid        0.392x  parity True but slower
grouped row scoring         0.993x  exact but slower
prediction-cell hybrid      0.816x  not row-exact
finalize strategy eval     54.792x  parity False
attempt analysis           139.656x parity False

Current Runtime Chokepoints After Python Optimization

These were recorded after the accepted Python/runtime cleanup work:

Hotspot	Time
`_run_finalize_chunk_worker`	`38.790s`
`build_oos_validation`	`38.464s`
`_dense_arrays_to_prepared_outcomes`	`28.974s`
`build_event_batch_for_variant`	`14.715s`

Interpretation:

Even after Python cleanup, object-heavy finalize/build stages remained expensive.
The accepted Rust ports targeted the same kind of repeated numeric and materialization work.
This explains why native stage ports could still show large wins after Python had already been optimized.

Current ORB Rust Profile Evidence

Current ORB profile evidence:

family: orb
representative candidate: orb_tf1_or1_dl1_rtdirect_sigfirst_breakout_only_cc0_body0p0_scaled_candle_extreme_tgttouch

Profile passes:

Pass	Scheduler	Workers	Pinning	Session block	Param block	Load/prep	Feature build	Signal eval	Combo eval	Total wall
`pass0_baseline`	`session_major`	`8`	`none`	`1`	`8`	`3087us`	`1us`	`499us`	`2572us`	`3073us`
`pass1_reprofile`	`hybrid_blocked`	`16`	`none`	`24`	`8`	`1233us`	`0us`	`472us`	`760us`	`1233us`
`pass2_reprofile`	`session_major`	`16`	`none`	`1`	`64`	`22453us`	`0us`	`488us`	`21964us`	`22453us`

Profile interpretation:

pass0_baseline identified combo evaluation as the dominant hotspot.
pass1_reprofile reduced combo evaluation from 2572us to 760us on the realistic shape.
pass2_reprofile intentionally used a denser stress shape, so combo evaluation rose to 21964us.
Signal evaluation stayed around 472us to 499us, much smaller than dense combo evaluation.

Visual ORB hotspot chart:

pass0 combo eval   2572us | ##########
pass1 combo eval    760us | ###
pass2 combo eval  21964us | ########################################################################################

Scale: one # is approximately 250us.

Implementation-Level Design Facts

This section summarizes the implementation facts behind the benchmark claims.

Market Data Structure

Current Rust market data is columnar:

OhlcvFrame
- symbol: String
- day_keys: Vec<u32>
- minute_of_day: Vec<u16>
- open: Vec<f32>
- high: Vec<f32>
- low: Vec<f32>
- close: Vec<f32>
- volume: Vec<f32>
- sessions: Vec<SessionRange>

SessionRange
- day_key: u32
- start: usize
- end: usize

RsiPlane
- period: usize
- values: Vec<f32>

ResidentFrame
- timeframe_minutes: usize
- frame: Arc<OhlcvFrame>
- day_to_session: Arc<HashMap<u32, usize>>
- rsi_planes: Vec<Arc<RsiPlane>>

ResidentBundle
- symbol: String
- years: usize
- frames: Vec<ResidentFrame>

Performance impact:

Hot loops use compact arrays.
Session boundaries are known.
Timeframe and RSI data stay resident.
Memory bandwidth and cache locality are much better than Python row objects.

Binary Cache Contract

The market cache stores:

magic bytes: RBCACHE1
cache version: 1
symbol string
row count
session count
database length
database mtime seconds
day_keys
minute_of_day
open
high
low
close
volume
session ranges

Performance impact:

Later runs avoid full SQLite row conversion.
Cache validity is tied to DB size and modification time.
Cached bytes map directly back to typed vectors.

SQLite Read Settings

The Rust database loader applies read-oriented settings:

PRAGMA query_only = ON;
PRAGMA temp_store = MEMORY;
PRAGMA cache_size = -200000;
PRAGMA mmap_size = 268435456;

Performance impact:

The DB path is treated as read-only.
SQLite page/cache behavior is tuned for read-heavy backtests.
The loader avoids unnecessary write-path behavior.

ORB Entry Preparation

Current ORB-like flow:

prepare_entries
  -> prepare_orb_entry
  -> prepare_orb_signal_entry
  -> materialize_orb_prepared_entry
  -> evaluate_entry_combo_block

Prepared entry fields include:

session identity
signal day and minute
decision minute
entry minute
entry price
side
stop kind
base risk
future start index
future end index
session close price
execution model
tradeability metadata

Performance impact:

Signal discovery is paid once per session/candidate.
SL/RR evaluation consumes prepared numeric state.
Full trade records are not required for every leaderboard row.

Combo Evaluation Scratch Buffers

The current evaluator uses reusable scratch:

WorkerScratch
- stop_prices: Vec<f32>
- target_prices: Vec<f32>
- risk_points: Vec<f32>
- metrics: Vec<TradeMetric>
- pending_masks: Vec<u64>

Performance impact:

Fewer allocations.
Contiguous arrays for stop/target thresholds.
Compact pending state.
Same scratch reused across sessions/jobs.

Pending Mask Logic

Current logic:

For each combo block:
  pending_mask = all bits set

For each future bar:
  stop_mask = compare(bar stop-side price, stop_prices) & pending_mask
  clear stopped bits

  target_mask = compare(bar target-side price, target_prices) & pending_mask
  clear target-hit bits

  if all pending masks are zero:
    break early

Performance impact:

Resolved combos stop consuming work.
Up to 64 combos fit in one pending word.
Stop/target checks feed directly into mask operations.

AVX2 Compare Path

The current compare mask has two paths:

if AVX2 detected:
  compare 8 f32 thresholds at a time
  convert comparison result to bitmask
else:
  scalar loop over thresholds

AVX2 operations used:

_mm256_set1_ps
_mm256_loadu_ps
_mm256_cmp_ps
_mm256_movemask_ps

Performance impact:

Threshold checks are vectorized.
The vector result becomes the same bitmask format used by pending masks.
Machines without AVX2 still run the scalar path.

Persistent Worker Pools

Current worker-pool shape:

PersistentPools
- one pool per worker count for pinning=none
- one pool per worker count for pinning=physical
- jobs submitted through channels
- worker threads stay alive until pool drop

Performance impact:

Benchmarks and search jobs avoid repeated thread creation.
Tuning matrices can compare worker counts consistently.
Work can be scheduled by session blocks, combo blocks, or per-param jobs.

Scheduler Modes

Current scheduler modes:

Scheduler	Shape	Use
`session_major`	split sessions across jobs, evaluate combos inside each session	good when entries are many and combo block batching is useful
`param_major`	split combo blocks, iterate entries inside combo block	useful for combo-heavy workloads
`hybrid_blocked`	split both sessions and combo blocks	useful when both dimensions need balancing
`thread_per_param`	distribute narrow combo work across workers	useful for certain ORB/MTF grids and defaults

ORB Family Coverage In Rust

Current ORB-like families:

orb
orb_mtf
orb_human
orb_mtf_human

Current strategy dimensions:

Dimension	Current Rust support
Timeframes	single-timeframe and multi-timeframe
Opening range	opening range candle count, OR timeframe
Deadline	breakout deadline candles
Retest logic	direct, single retest, double retest
Signal pattern	first breakout only, opposite reversal allowed, opposite reversal required
Confirmation	close confirmation, confirm chain, higher timeframe confirmation
Execution	legacy same-bar, close-confirm next open, resting limit at level
Stop	scaled candle extreme, opposite range, fixed points
Target	touch, next close confirm, confirming bar close
Carry	session exit
Context	gap bucket, open vs previous range, ORB color, breakout side, breakout extension, reversal observed

Performance relevance:

These variants share resident market data.
They share entry preparation.
They share combo-grid evaluation.
They share scheduler tuning and runtime baseline output.
Adding variants does not require returning to Python object-heavy loops.

Code-Level Mapping From Bottleneck To Rust Mechanism

Old bottleneck category	Current Rust mechanism	Evidence quality	Performance effect
DB row conversion per run	read-only loader plus binary cache	direct measured, implementation verified	base loading `25.192x`
Repeated timeframe rebuilds	cached session-aware `OhlcvFrame` resampling	direct measured, implementation verified	resample `33.297x`
Python range/event loops	typed Rust range discovery loops	direct measured	range discovery `92.306x`
Repeated signal work per SL/RR	`prepare_entries` before combo eval	implementation verified, inferred ORB effect	removes candidate x combo signal repetition
Future scan per combo row	`evaluate_entry_combo_block` scans many combos together	implementation verified	fewer scans, better cache locality
Python dict/list reconstruction	summary-first Rust output and selected trade materialization	direct measured in dense prepared/event batch	dense/event stages `68.817x` and `152.802x`
Stop/target branch churn	pending masks and early exit	implementation verified	resolved combos are skipped
Scalar threshold checks	AVX2 compare mask with scalar fallback	implementation verified	8 thresholds compared per vector on AVX2 machines
Repeated thread creation	`PersistentPools`	implementation verified	lower scheduling overhead for benchmark/search jobs
One fixed scheduler	tuned scheduler matrix and baselines	current ORB profile	selects better worker/scheduler/block settings by workload

Claim Ledger

Claim	Status	Evidence
Rust data loading is much faster than Python loading	proven for benchmark stage	`36.899s -> 1.465s`, exact parity true
Rust resampling is much faster than Python resampling	proven for benchmark stage	`60.053s -> 1.804s`, all symbol/timeframe parity true
Rust range discovery reached about 100x speedup	proven for benchmark stage	`118.732s -> 1.286s`, `92.306x`, parity true
Rust finalize event batch crossed 100x	proven for benchmark stage	`21.037s -> 0.138s`, `152.802x`, parity true
Dense prepared Rust path showed very large speedup	proven as hotspot upper-bound	`68.817x` vs production prepared path, `310.187x` reducer-summary
Current ORB combo eval has measurable hotspots	proven in ORB profile	`pass0/pass1/pass2` timing table
Rust ORB uses typed resident arrays	implementation verified	`OhlcvFrame` field list
Rust ORB uses prepared entries and batched combos	implementation verified	flow and scratch buffer facts
Rust ORB uses AVX2 where available	implementation verified	AVX2 function family
Full ORB end-to-end old Python vs Rust is 100x	not proven	direct matching artifact not found
Every Rust attempt was accepted	false	rejected experiments table

What Is Directly Measured, Inferred, And Missing

Directly measured:

Six accepted Rust stage ports with Python time, Rust time, speedup, and parity status.
Current ORB Rust profile passes and scheduler choices.
Python/runtime optimization counts and before/after timings.
Rejected experiment timings and reasons.

Inferred:

Exact shape of the old Python ORB loop.
Which old Python functions map to which Rust functions.
Exact end-to-end full ORB speedup.
The detailed order in which individual Rust ORB implementation functions were written.

Missing:

Old Python ORB implementation.
Git commit history from the migration.
Direct old-Python-ORB run artifact on the same fixture as current Rust ORB.
A full end-to-end ORB benchmark matrix comparing old Python and current Rust.

Why The 100x-Class Claim Is Still Technically Fair

It is fair if stated as:

Several heavy benchmarked stages that ORB-like backtesting depends on reached 100x-class Rust speedups, with accepted stage evidence up to 152.802x.

It is not fair if stated as:

Every full ORB backtest is proven to be 100x faster end to end.

Correct wording for future reports:

“100x-class stage speedups were measured.”
“The current Rust ORB architecture applies the same principles: resident typed data, prepared entries, batched combo evaluation, masks, SIMD, and scheduler tuning.”
“A direct end-to-end old Python ORB comparison is still missing.”

Historical Timeline From Available Evidence

2026-04-25
  Chunk scheduling benchmark work was active.

2026-04-26
  Finalize optimization improved end-to-end modestly and finalizer substantially.

2026-04-27
  Python/runtime optimization phase:
  - 1m base-loading contract plus in-memory resample
  - RR/stop batching
  - coordinator/finalize checkpoint and OOS cleanup
  - dense reducer redesign

2026-04-28
  Finalize/OOS cleanup continued:
  - prepared outcome cleanup
  - final input cache v2
  - OOS probability map cleanup
  - dense arrays to prepared cleanup
  - OOS rank prune
  - split finalize cache
  - dense prepared Rust benchmark captured

2026-05-01
  Rust stage-port benchmark suite captured:
  - base data loading
  - higher-timeframe resample
  - range discovery
  - chunk analytics
  - finalize event batch
  - rejected Rust/DuckDB attempts documented

2026-05-27
  Current Rust backtester/ORB artifacts inspected.

2026-06-04
  Benchmark record consolidated from available evidence.

Reproduction Notes

For future benchmark runs:

Rebuild the Rust binary immediately before testing.
Confirm the executable supports the target ORB family before timing it.
Keep cache-warm and cache-cold runs separate.
Preserve the exact fixture: database, symbol, date range, timeframe, ORB parameters, stop/RR grid, and output requirements.
Compare both timing and parity before accepting a speedup claim.

Important stale-binary note:

A stale binary can launch successfully and still reject a newly added family as unsupported.
Before assuming a family is broken, confirm the executable was rebuilt and supports that family.
Toolchain or launch-policy errors should be treated separately from benchmark correctness.

Profile metrics to compare after a new ORB change:

load_prep_us
feature_build_us
signal_eval_us
combo_eval_us
total_wall_us
selected scheduler
worker count
session block
param block

Gap Closure Plan

To make this a complete historical migration record:

Recover the old Python ORB implementation.
Identify one exact old Python ORB command and fixture.
Run or recover old Python timing for cold load, warm load, signal prep, combo evaluation, and output writing.
Run current Rust on the same fixture with the same output requirements.
Add a direct end-to-end table with Python wall time, Rust wall time, speedup, parity checks, and output row counts.
Add function-level mapping between the old Python implementation and the current Rust implementation.

Final Takeaway

The useful record is:

The measured speedups.
The visual comparison.
The benchmark card details.
Accepted Python optimizations.
Accepted Rust stage ports.
Rejected or conditional experiments.
Current ORB profile evidence.
Implementation-level design facts.
Claim boundaries.
Remaining gaps.

The most accurate final statement is:

The Rust migration produced measured 100x-class speedups in several heavy backtesting stages, including 92.306x for range discovery and 152.802x for finalized event batch building. The current Rust ORB engine uses the same core performance design: typed resident data, cached resampling, prepared entries, batched SL/RR evaluation, pending masks, AVX2 compares, persistent workers, and profiled scheduler tuning. A direct full old-Python-ORB versus current-Rust-ORB end-to-end benchmark is still missing, so the 100x claim should be stated as stage-level rather than universal full-run proof.

ORB Python-To-Rust Speedup Record

Scope And Limitations

Executive Summary

Evidence Inventory

Visual Architecture Change

Current Rust ORB Pipeline

Optimization Count Chart

Speedup Layer 1: Within Python Changes

Python Phase Visual

What Changed Inside Python

Python Lessons That Carried Into Rust

Python Phase Takeaway

Accepted Python And Runtime Optimizations

Accepted Rust Stage Ports

Python Time vs Rust Time Chart

Speedup Layer 2: Within Rust Changes

Rust Phase Visual

Current ORB Rust Pass-To-Pass Tuning Evidence

What Changed Inside Rust

Rust Phase Takeaway

Benchmark Card 1: Base Data Loading

Benchmark Card 2: Higher-Timeframe Resample

Benchmark Card 3: Range Discovery

Benchmark Card 4: Chunk Analytics Build

Benchmark Card 5: Dense Prepared Reconstruction

Benchmark Card 6: Finalize Event Batch Build

Rejected Or Conditional Experiments

Current Runtime Chokepoints After Python Optimization

Current ORB Rust Profile Evidence

Implementation-Level Design Facts

Market Data Structure

Binary Cache Contract

SQLite Read Settings

ORB Entry Preparation

Combo Evaluation Scratch Buffers

Pending Mask Logic

AVX2 Compare Path

Persistent Worker Pools

Scheduler Modes

ORB Family Coverage In Rust

Code-Level Mapping From Bottleneck To Rust Mechanism

Claim Ledger

What Is Directly Measured, Inferred, And Missing

Why The 100x-Class Claim Is Still Technically Fair

Historical Timeline From Available Evidence

Reproduction Notes

Gap Closure Plan

Final Takeaway

Comments