Skip to content

Cache and Scratch #1138

Open
Open

Description

@Affie

So I really like:

  • Cache: operational memory used by default solver routines during compute in hot-loops etc for maximizing performance of source code. E.g. storing calculated gradients, hypothesis recipes, or ode-solutions. Good solver will keep this isbitstype == true. Any non-standard in-memory objects should instead use the "scratch" field.
  • Scratch: Custom in-memory location generally associated with the loading or saving of objects, for holding case-by-case in-memory values that accelerate the compute tasks. "Scratch" is available for custom designs of e.g. factors whereby the developer wishes to manually craft in-place objects to optimize compute operations - e.g. loading lidar blobs from storage into a factor scratch as well as faster rigid transform registration compute techniques. Up to user to be good developer and eventually get .scratch to isbitstype == true, but doesn't have to be on day one. In contrast, solver default routines should instead use the "cache" field.

This means that any time a sampler or residual function runs, the user will have access to the CalcFactor object's mutable memory:

  • calcfactor.scratch: here user/developer does their custom work to make the machine go. Also whatever the user setup on factor load/add.
  • calcfactor.cache: in-place solver stored values such as gradients or whatever else happens per default.

Remember the solver design is going down road of always creating a new CalcFactor object for each independent thread. I.e. calcfactor.* only lasts as long as the thread/hot-loop used to compute with it.

Then there is also the issue of parching, e.g. load a FactorCompute from DBDFG or FileDFG such as lidar registration where .laz files representing two point clouds are stored as blobs. Then the preambleScratch function would run once and a user is responsible for something like scratch.pc1 = getBlob(parchedfactor.pointcloud1blobid,...). We have a first example of this, let me see if I can find... yes all the ObjectAffordSubcloud stuff: https://github.com/JuliaRobotics/Caesar.jl/blob/9df2df54d4a1bc271afafd5730d35c95af4c67f5/src/objects/ObjectAffordanceSubcloud.jl#L241

Originally posted by @dehann in #1127 (comment)

Activity

  1. added this to the v0.0.x milestone on Jun 19, 2025
  2. modified the milestones: v0.0.x, v0.28.0 on Jul 10, 2025
  3. modified the milestones: v0.28.0, v0.29.0 on Sep 2, 2025
  4. Affie commented on Apr 20, 2026

    @Affie
    MemberAuthor

    rebuildFactorCache! is currently coupled to solverParams so removing SolvarParams (#1205) influences how caching works.

  5. Affie commented on Apr 20, 2026

    @Affie
    MemberAuthor

    Option 1: "Always There" (Implicit Lifecycle)
    The cache is automatically populated the moment a factor is created (e.g., via addFactor!) or loaded from disk. It permanently occupies RAM for the entire lifespan of the factor in the graph, regardless of whether it is actively being used in a solve.

    Option 2: "Built Just-In-Time" (Explicit Lifecycle)
    The factor initializes with an empty cache. The solver is strictly responsible for building the cache during a "warm-up" phase right before inference. This allows the solver to actively manage RAM—building caches only for the active subgraph and discarding them when no longer needed, while the core factor data remains safely in the graph.

  6. Affie commented on Apr 20, 2026

    @Affie
    MemberAuthor

    We need to finalize the verb to build the cache. Currently rebuild as in rebuildFactorCache!.
    (WIP) Some other options:

    • warmup
    • The julia get! function, get!(f::Function, collection, key) or get!(collection, key, default)
    • hydrateCache!
    • inflate!
    • build!
  7. dehann commented on Apr 20, 2026

    @dehann
    Member

    Caveats:

    • Data Level 2 graph retrieval is not exported or hinted yet in DFG v1.0-alpha

    Formalizing DFG "Data Level 4" -- i.e. {Variable,Factor}Compute via UX

    UX 1 - Local Data Wrangling (likely a power user)

    using DFG
    
    # blank interface signatures are called, but rebuildCache implementations are blank
    # no cache has been inflated but is ready to do so
    # A Data Level 3 object
    fg = loadDFG("/tmp/somegraph.tar.gz")
    
    using RoME
    
    # this triggers JIT/lazy rebuildCache 
    valfromcache = _getSomethingFromCache(fg, id) 

    UX 2 - Remote Data Wrangling (likely an newbie)

    using NavAbilitySDK
    
    # blank interface signatures are called, but rebuildCache implementations are blank
    # no cache has been inflated but is ready to do so
    # a Data Level 3 object
    fg = NavAbilityDFG(..., :somegraph; auth, ...)
    
    using RoME
    
    # this triggers JIT/lazy rebuildCache (user perceives slower performance)
    valfromcache = _getSomethingFromCache(fg, id) 

    UX 3 - Local Factor Dev (blind customer DX)

    using Caesar
    
    # interface signatures are called, and rebuildCache's are overloaded for dispatch
    # A Data Level 3 object / "Always There"
    fg = loadDFG("/tmp/somegraph.tar.gz")
    
    # now ALL CACHE has been inflated to maximum - verb TBD
    inflate!(fg) # as per @info hint if missed during first solveGraph!
    
    # and this works absolute minimum solve time, zero JIT on first execute (w precompile/preworkloads per Caesar/RoME/IIF/etc)
    @time solveGraph!(fg)
    
    # always print stats with breakdown on timing for inflateCaches vs precompile vs solve

    UX/DX 4 - Local Factor Dev, using NavAbility API data (customer DX)

    using NavAbilitySDK
    using RoME
    
    # see this as a borrow/reference with same API -- use signatures
    fg = NavAbilityDFG(..., :somegraph; auth, ...)
    
    # rebuildCache was already done
    fac = getFactor(fg, :lbl) # this won't work without `using RoME/IIF/Caesar`
    # lazy inflate will take time because it needs API calls, e.g. lidar registration factor (fetch .laz blobs)
    valfromcache = _getSomethingFromCache(fac) 

    UX 5 - Performance DX

    using NavAbilitySDK
    using DistributedFactorGraph
    
    # blank interface signatures are called, but rebuildCache implementations are blank
    # no cache has been inflated but is ready to do so
    # a Data Level 3 object
    rfg = NavAbilityDFG(..., :somegraph; auth, ...)
    
    # sync to blank (Johan help??)
    # blank interface signatures are called, but rebuildCache implementations are blank
    # no cache has been inflated but is ready to do so
    # a Data Level 3 object
    lfg = LocalDFG()
    sync!(lfg, rfg; where=...)
    
    let
      using RoME
      # similar to UX 1, poweruser has reason for local scope, maybe Distributed.jl addprocs, etc.
      solveGraph!(lfg) 
    end

    UX 6 - Offload Solve

    using NavAbilitySDK
    
    # blank interface signatures are called, but rebuildCache implementations are blank
    # no cache has been inflated but is ready to do so
    # a Data Level 3 object
    fg = NavAbilityDFG(..., :somegraph; auth, ...)
    
    # launch remote action
    tsk = solveGraphAsync!(fg)
    
    # just for info, sync should warn and hint to maybe try `solveGraphAsync!`
    solveGraph!(fg) # because ::NavAbilityDFG
  8. Affie commented on Apr 21, 2026

    @Affie
    MemberAuthor

    I didn't know from just reading the first comment to what Scratch referred and after an explanation (meaning it might be the wrong noun) it is the "userCache" in IIF, called as:

    • userCache = preambleCache(dfg, Xi, usrfnc)
    • In CalcFactor the field is just called cache.
    • _createCCW also userCache and then also dummyCache
    • CalcFactorNormSq calls this cache = ccwl.dummyCache

    From the current use in IIF, SolverCache and UserCache are also options.

    The name should reflect: “user cache/preallocs for use in factor residual calculations hot loops“.

    Nouns for cahche/preallocs part:

    • Workspace
    • Buffer
    • Cache - SciML calls similar objects cache, see PreallocationTools.
    • Preallocs

    User observation Lifecycle (all optional):

    • Deserialization / Unmarshalling: unpack observation
    • Materialization / Hydration / Resolution / Binding: hydrate observation (Lazy Loading vs. Eager Fetching)
    • Thread-Local Storage / Preallocated Workspace / Scratchpad Memory / Ephemeral State: build the user cache that is available for residual functions

    Building the solver cache is separate from the observation and a solver specific feature not influenced by user domain.

  9. Affie commented on Apr 29, 2026

    @Affie
    MemberAuthor

    In a conversation with Dehann using Stash also came up. stash is currently used as a verb, but I'm open to changing stashBlobproviders! or using Stash as a noun. However, from the noun alone I'm not sure where it will be used and that kind of says it is not the correct noun.

  10. Affie commented on May 2, 2026

    @Affie
    MemberAuthor

    I’m leaning towards prepareCache (to stay close to preambleCache) or buildObservationCache for building the user object to use in “Thread-Local Storage / Preallocated Workspace / Scratchpad Memory / Ephemeral State: build the user cache that is available for residual functions”

    I also found the original preambleCache issue here: JuliaRobotics/IncrementalInference.jl#1462
    The difference between my suggested observation cache object and the preambleCache object is that the observation cache object should be thread safe and prepareCache will be called for each thread.

  11. modified the milestones: v0.29.0, v0.30.0 on May 7, 2026
  12. Affie commented on Jul 27, 2026

    @Affie
    MemberAuthor

    Similar to the question on a Dict of FactorDFG.state by state label (solve key). I have a need for cache to be keyed to enable reuse (depending on where we want the cache to be).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions