Repository navigation
Expand the atlas to 3,122 nodes (dataset 0.9.0) - #30
Merged
Merged
Conversation
Adds about 1,490 nodes and 4,800 edges researched from roadmap.sh, the Thoughtworks Technology Radar, the CNCF landscape, awesome-lists and Wikipedia lists: old and legacy tech, newer tools, and overlooked fields. - 3 new top-level fields: Programming Languages, Data Science & Analytics, Enterprise & Business Software. - 8 new sub-fields: Generative AI & LLMs, Applied Cryptography, Observability & SRE, Mainframes & Legacy, CMS & E-commerce, Audio & Music, Automotive & Aerospace, Geospatial. Existing nodes in those areas moved into them. - Expansion lives in scripts/expand/atlas_2000*.py (compact node DSL in _dsl.py) and merges into the seed through apply_batch. - Famous apps: not_in_atlas labels that now exist move into the stacks. - Scanner: new node names are matched by package name, so wrong matches are blocked in package_map.json (JAXB "bind", codehaus "mojo", Google Dagger, the "grep" crate, Python-Markdown, Dart mockito, Ruby mocha) and correct new detections are added to the evaluation ground truth. - Budgets: graph payload 239 KB and entry JS 168 KB gzip; see docs/performance.md for the light-index follow-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EMFEELjnuQWbprj8fXp13r
Adds 441 nodes and about 1,540 edges on top of 0.8.0, for 3,122 nodes and 9,834 edges in all: - Open source from tech companies' portals: Netflix, Uber, Meta, Google, LinkedIn, Twitter, Spotify, Shopify, Stripe, Lyft, Microsoft, AWS, Apple, Alibaba, Tencent, Baidu, Cloudflare and Vercel. - Databases, commerce and site builders, low-code tools, CI/CD, Kubernetes tooling and observability missing from the atlas. - Everyday developer tools: editors, shells, CLI utilities, build and package tools, linters, test, docs, diagram and chart tools. - Glossary terms: design patterns, distributed-systems and data terms, concurrency, data structures, web and practice terms, and laws such as Conway's and Hyrum's. New data lives in scripts/expand/atlas_3000*.py with its own runner, which also rejects names that clash with existing nodes. Sorbet and Amundsen move from not_in_atlas into the Stripe and Lyft stacks. Scanner: block the Ruby "stoplight" gem and the "wiremock" crate, which are different projects; add correct new detections to ground truth. Layout: LANDSCAPE 1.6 -> 1.5 so every field heading fits at 1440x900. Graph payload budget raised to 273 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EMFEELjnuQWbprj8fXp13r
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Rush · project thread
What this does
This is the big data expansion before the public launch, in two commits. Dataset 0.9.0 has 3,122 nodes (up from 1,194), 9,834 edges (up from 3,517) and 40 domains (up from 29).
The node list came from roadmap.sh, the Thoughtworks Technology Radar (vols. 33 and 34), the CNCF landscape, awesome-lists and Wikipedia's lists of languages and software. It covers old and legacy tech (COBOL, z/OS, CICS, Fortran, Perl, Delphi), newer tools (MCP servers, agent frameworks, vLLM, uv, Biome), and fields the atlas was missing.
New fields
New nodes by field (commit 1)
Programming Languages 128, Cloud/DevOps 117, Databases 98, Testing & Tooling 81, Data Science 76, GenAI 74, Frontend 73, AI/ML 72, Backend 69, Systems 63, Cybersecurity 61, Enterprise 51, Observability 40, Networks 38, Cryptography 37, Audio & Music 34, Mainframes 32, Theory 32, Auto & Aero 29, Geospatial 26, plus 1 to 25 each in 16 other fields.
How it's built
scripts/expand/atlas_2000*.pyandatlas_3000*.py, which use a compact node and edge DSL in_dsl.py. Each runner's--checkflag reports problems without writing anything. The 3,000 runner also rejects any name or alias that clashes with an existing node.implements,used_for,supportsandenablesedges that point at a technology are downgraded torelated_to, so the validator's endpoint rules hold.LANDSCAPEwent from 1.6 to 1.5 in commit 2, so every field heading still fits at 1440×900.not_in_atlasthat are now atlas nodes moved into the stacks (memcached, Vitess, Hive, Presto, Pinot, ScyllaDB, Sorbet, Amundsen…).accuracy_pass.pywas retired, because OpenSearch is now a node.Scanner side effect
The scanner matches packages to nodes by name, so the new nodes changed what it detects:
nullentries inpackage_map.json:*.xml.bindmatched BIND DNS.org.codehaus.mojomatched the Mojo language.com.google.dagger(the DI library) matched Dagger CI.grepcrate matched GNU grep.markdownmatched the Markdown format.mockitoand Rubymochamatched the Java and JS libraries of the same name.stoplightgem (a circuit breaker) matched the Stoplight API platform, and thewiremockcrate matched the Java WireMock.added_in_reviewand their evidence. Examples are pydantic, tokio, Sidekiq and PostCSS.Performance
The graph payload roughly doubled:
api/v1/graph.jsongzip* The 0.7.0 doc figure is older; main was already near the 166 KB budget.
docs/performance.md.Testing
validate_dataset.py --require-targets: valid, with 8 warnings.pytest: 270 passed, 1 skipped. One discovery test used "Elysia" as a made-up framework name; Elysia is a real node now, so the test uses an invented name.vitest: 160 passed.tscandnpm run sizeare clean.npm run e2e): 11 passed.https://claude.ai/code/session_01EMFEELjnuQWbprj8fXp13r
Generated by Claude Code