Skip to content

Expand the atlas to 3,122 nodes (dataset 0.9.0) - #30

Merged
rushilrawat merged 2 commits into
mainfrom
data/atlas-2000
Nov 20, 2026
Merged

rushilrawat merged 2 commits into
mainfrom
data/atlas-2000

Conversation

@rushilrawat

@rushilrawat rushilrawat commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Requested by Rush · project thread

What this does

This is the big data expansion before the public launch, in two commits. Dataset 0.9.0 has 3,122 nodes (up from 1,194), 9,834 edges (up from 3,517) and 40 domains (up from 29).

  • Commit 1 (0.8.0, 2,681 nodes): new fields, old and new tech, overlooked areas.
  • Commit 2 (0.9.0, 3,122 nodes): 441 more nodes. These are open source from tech companies' portals (Netflix, Uber, Meta, Google, LinkedIn, Twitter, Spotify, Shopify, Stripe, Lyft, Microsoft, AWS, Apple, Alibaba, Tencent, Baidu, Cloudflare, Vercel), plus databases, low-code and commerce platforms, CI/CD, Kubernetes and observability tools, everyday developer tools (editors, shells, build, lint, test, docs, diagrams, charts), and Wikipedia glossary terms (design patterns, distributed-systems and data terms, concurrency, data structures, laws like Conway's and Hyrum's).

The node list came from roadmap.sh, the Thoughtworks Technology Radar (vols. 33 and 34), the CNCF landscape, awesome-lists and Wikipedia's lists of languages and software. It covers old and legacy tech (COBOL, z/OS, CICS, Fortran, Perl, Delphi), newer tools (MCP servers, agent frameworks, vLLM, uv, Biome), and fields the atlas was missing.

New fields

  • Top-level fields: Programming Languages, Data Science & Analytics, Enterprise & Business Software.
  • Sub-fields: Generative AI & LLMs, Applied Cryptography, Observability & SRE, Mainframes & Legacy, CMS & E-commerce, Audio & Music, Automotive & Aerospace, Geospatial.
  • Existing nodes in those areas moved into the new sub-fields. For example, the LLM concepts, LangChain, vLLM and Ollama moved into GenAI, and Prometheus, Grafana and OpenTelemetry moved into Observability.

New nodes by field (commit 1)

Programming Languages 128, Cloud/DevOps 117, Databases 98, Testing & Tooling 81, Data Science 76, GenAI 74, Frontend 73, AI/ML 72, Backend 69, Systems 63, Cybersecurity 61, Enterprise 51, Observability 40, Networks 38, Cryptography 37, Audio & Music 34, Mainframes 32, Theory 32, Auto & Aero 29, Geospatial 26, plus 1 to 25 each in 16 other fields.

How it's built

  • All new data is in scripts/expand/atlas_2000*.py and atlas_3000*.py, which use a compact node and edge DSL in _dsl.py. Each runner's --check flag reports problems without writing anything. The 3,000 runner also rejects any name or alias that clashes with an existing node.
  • Edges:
    • Every new node has at least 2 typed edges, as the validator requires.
    • implements, used_for, supports and enables edges that point at a technology are downgraded to related_to, so the validator's endpoint rules hold.
    • Duplicates of existing nodes (dbt, OIDC, WebAuthn, BLE, ELT…) were merged into the existing node instead of being added.
  • Accuracy:
    • Wikipedia titles are only set where the article is about that exact subject.
    • Lifecycle states (legacy, deprecated) follow each project's own announcements.
  • Layout:
    • The radial layout supports only two levels, so Audio & Music and Automotive & Aerospace hang directly off Computer Engineering.
    • The new top-level fields have colours, short headings and places in the layout rows.
    • LANDSCAPE went from 1.6 to 1.5 in commit 2, so every field heading still fits at 1440×900.
  • Famous apps: labels in not_in_atlas that are now atlas nodes moved into the stacks (memcached, Vitess, Hive, Presto, Pinot, ScyllaDB, Sorbet, Amundsen…).
    • The OpenSearch rule in accuracy_pass.py was retired, because OpenSearch is now a node.

Scanner side effect

The scanner matches packages to nodes by name, so the new nodes changed what it detects:

  • Wrong matches, now blocked with null entries in package_map.json:
    • JAXB *.xml.bind matched BIND DNS.
    • org.codehaus.mojo matched the Mojo language.
    • com.google.dagger (the DI library) matched Dagger CI.
    • The grep crate matched GNU grep.
    • PyPI and pub markdown matched the Markdown format.
    • Dart mockito and Ruby mocha matched the Java and JS libraries of the same name.
    • The Ruby stoplight gem (a circuit breaker) matched the Stoplight API platform, and the wiremock crate matched the Java WireMock.
  • Correct new detections: 279 of them, each checked against the manifest line that declares it, were added to the evaluation ground truth with added_in_review and their evidence. Examples are pydantic, tokio, Sidekiq and PostCSS.
  • Optional: self-detections such as Streamlit in the Streamlit repo were marked optional.

Performance

The graph payload roughly doubled:

0.7.0 0.9.0
api/v1/graph.json gzip 99 KB 252 KB
search index gzip (loaded on first search) 61 KB 148 KB
entry JS gzip (embeds apps.json) 137 KB* 167 KB

* The 0.7.0 doc figure is older; main was already near the 166 KB budget.

  • Budgets were raised to 273 KB for the graph and 168 KB for JS, and the change is documented in docs/performance.md.
  • This is past the point where the performance doc says to switch to a light index loaded first, with node details fetched on demand. That should be the next performance PR.
  • The overview still renders cleanly at 3,122 nodes (screenshot checked at 1440×900).

Testing

  • validate_dataset.py --require-targets: valid, with 8 warnings.
    • 7 are alias overlaps between distinct nodes (CI, CD, SaaS, SBOM, Swagger, Android, AI Safety).
    • 1,005 nodes have fewer than 3 typed edges. That is a warning-level target; the 2-edge floor is met.
  • Backend pytest: 270 passed, 1 skipped. One discovery test used "Elysia" as a made-up framework name; Elysia is a real node now, so the test uses an invented name.
  • Frontend vitest: 160 passed. tsc and npm run size are clean.
  • Playwright smoke test (npm run e2e): 11 passed.
  • Overview visual-proof suite: 21 of 22 passed. The failure also happens on main:
    • "guided path follows real nodes" fails there every time.
    • Separately, "connection legend…selected curve" is flaky on main too (1 run in 6).

https://claude.ai/code/session_01EMFEELjnuQWbprj8fXp13r


Generated by Claude Code

Adds about 1,490 nodes and 4,800 edges researched from roadmap.sh, the
Thoughtworks Technology Radar, the CNCF landscape, awesome-lists and
Wikipedia lists: old and legacy tech, newer tools, and overlooked fields.

- 3 new top-level fields: Programming Languages, Data Science &
  Analytics, Enterprise & Business Software.
- 8 new sub-fields: Generative AI & LLMs, Applied Cryptography,
  Observability & SRE, Mainframes & Legacy, CMS & E-commerce, Audio &
  Music, Automotive & Aerospace, Geospatial. Existing nodes in those
  areas moved into them.
- Expansion lives in scripts/expand/atlas_2000*.py (compact node DSL in
  _dsl.py) and merges into the seed through apply_batch.
- Famous apps: not_in_atlas labels that now exist move into the stacks.
- Scanner: new node names are matched by package name, so wrong matches
  are blocked in package_map.json (JAXB "bind", codehaus "mojo", Google
  Dagger, the "grep" crate, Python-Markdown, Dart mockito, Ruby mocha)
  and correct new detections are added to the evaluation ground truth.
- Budgets: graph payload 239 KB and entry JS 168 KB gzip; see
  docs/performance.md for the light-index follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EMFEELjnuQWbprj8fXp13r
@rushilrawat rushilrawat self-assigned this Nov 20, 2026
Adds 441 nodes and about 1,540 edges on top of 0.8.0, for 3,122 nodes and
9,834 edges in all:

- Open source from tech companies' portals: Netflix, Uber, Meta, Google,
  LinkedIn, Twitter, Spotify, Shopify, Stripe, Lyft, Microsoft, AWS,
  Apple, Alibaba, Tencent, Baidu, Cloudflare and Vercel.
- Databases, commerce and site builders, low-code tools, CI/CD,
  Kubernetes tooling and observability missing from the atlas.
- Everyday developer tools: editors, shells, CLI utilities, build and
  package tools, linters, test, docs, diagram and chart tools.
- Glossary terms: design patterns, distributed-systems and data terms,
  concurrency, data structures, web and practice terms, and laws such
  as Conway's and Hyrum's.

New data lives in scripts/expand/atlas_3000*.py with its own runner,
which also rejects names that clash with existing nodes. Sorbet and
Amundsen move from not_in_atlas into the Stripe and Lyft stacks.

Scanner: block the Ruby "stoplight" gem and the "wiremock" crate, which
are different projects; add correct new detections to ground truth.
Layout: LANDSCAPE 1.6 -> 1.5 so every field heading fits at 1440x900.
Graph payload budget raised to 273 KB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EMFEELjnuQWbprj8fXp13r
@rushilrawat rushilrawat changed the title Expand the atlas to 2,681 nodes (dataset 0.8.0) Expand the atlas to 3,122 nodes (dataset 0.9.0) Nov 20, 2026
@rushilrawat
rushilrawat merged commit 122665a into main Nov 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants