diff --git a/medcat-trainer/docs/advanced_usage.md b/medcat-trainer/docs/advanced_usage.md index 85d7e500f..9fef36305 100644 --- a/medcat-trainer/docs/advanced_usage.md +++ b/medcat-trainer/docs/advanced_usage.md @@ -38,6 +38,16 @@ Modern project-admin UI uses dedicated endpoints: - `POST /api/project-admin/projects//clone/` - `POST /api/project-admin/projects//reset/` +## Project Group APIs + +- `GET/POST /api/project-groups/` +- `GET/PUT/PATCH/DELETE /api/project-groups//` + +`POST` accepts the shared project settings plus `annotators`, +`administrators`, and `create_associated_projects`. Prefer the +[Python client](client.md) helpers `create_project_group` / +`get_project_groups` for scripting. + ## Run a model over a project dataset Once a project has a dataset and a model (model pack, CDB + vocab, or a remote @@ -72,6 +82,21 @@ The Python client wraps this as `session.run_model(project)`. Only compatible projects should be combined (same underlying model configuration) when generating reports. +Concept Summary exclusion / low-F1 filtering in the UI is client-side over the +fetched report payload; there is no separate filter API. See +[Project Admin — Metrics reports](project_admin.md#metrics-reports). + +## Document text search API + +- `GET /api/search-documents/?dataset=&text=&page=` + +Returns DRF-style `{count, next, results}` pages (50 hits per page). Results +include document `id`, `name`, and `text`. Phrase search runs against the +dataset's indexed note text in the Solr collection named by +`DOCUMENT_SEARCH_COLLECTION`. When the dataset is not fully indexed or Solr is +unavailable, the endpoint falls back to a case-insensitive database contains +match. + ## Concept exploration and filter export Use the **Concepts** view (`/model-explore`) to: diff --git a/medcat-trainer/docs/annotator_guide.md b/medcat-trainer/docs/annotator_guide.md index 330335054..eabf4529d 100644 --- a/medcat-trainer/docs/annotator_guide.md +++ b/medcat-trainer/docs/annotator_guide.md @@ -7,11 +7,23 @@ predictions. ## 1) Document list -The left panel shows documents in the project dataset. +The left **Clinical Notes** panel shows documents in the project dataset. - Current document is highlighted. - Prepared documents (model predictions generated) are marked. - Submitted/validated documents are marked as complete. +- Drag the pane divider to resize the notes panel relative to the clinical text and concept sidebar. + +### Searching clinical notes + +Click the search icon in the Clinical Notes panel header to open search. + +| Mode | Behaviour | +|------|-----------| +| **Name** | Prefix match against document names already loaded in the browser. | +| **Text** | Full-text search across the dataset's clinical note bodies (Solr-backed). Matching spans are highlighted in the result snippets. Results are paginated; use **Load more** when available. | + +Text search indexes each dataset after upload. If indexing is incomplete or Solr is unavailable, the UI falls back to a slower database match. Admins can re-index older datasets from Django admin — see [Maintenance](maintenance.md) and [Django Admin](django_admin.md). ## 2) Clinical text diff --git a/medcat-trainer/docs/client.md b/medcat-trainer/docs/client.md index 19e77e6c8..1decdc658 100644 --- a/medcat-trainer/docs/client.md +++ b/medcat-trainer/docs/client.md @@ -57,9 +57,35 @@ run = session.run_model(project, wait=True, timeout=3600) print(run.status, run.prepared_docs, run.dataset_len) ``` +## Project groups + +Create one shared configuration and, by default, one annotation project per +annotator (named ` - `): + +```python +group = session.create_project_group( + name="Stroke IAA", + description="Inter-annotator agreement campaign", + annotators=[annotator], + administrators=[], # optional; the authenticated user is always an admin + dataset=ds, + modelpack=model_packs[0], + create_associated_projects=True, +) + +groups = session.get_project_groups() +``` + +Users, datasets, and model packs may be passed as wrapper objects or resolved +by name. Supply **either** `modelpack` **or** `concept_db` + `vocab`, not both. +With `create_associated_projects=True` (the default), at least one annotator is +required. + ## Common methods - `create_project(...)` +- `create_project_group(...)` +- `get_project_groups()` - `create_dataset(name, dataset_file)` - `create_user(username, password)` - `create_medcat_model(cdb, vocab)` @@ -82,9 +108,12 @@ print(run.status, run.prepared_docs, run.dataset_len) ## Notes -- `create_project` expects **either** `modelpack` **or** `concept_db + vocab`. +- `create_project` and `create_project_group` expect **either** `modelpack` + **or** `concept_db + vocab`. - Wrapper objects (`MCTDataset`, `MCTConceptDB`, etc.) can often be passed by object or resolved by name. +- See [Project Groups](project_group_admin.md) for the UI workflow these client + calls mirror. - `run_model` accepts a project object, id, or unique name. It runs the configured model over dataset documents that are still unprepared. Submitted documents, documents that already have annotations, and documents diff --git a/medcat-trainer/docs/django_admin.md b/medcat-trainer/docs/django_admin.md index cd8451131..1c22e0f36 100644 --- a/medcat-trainer/docs/django_admin.md +++ b/medcat-trainer/docs/django_admin.md @@ -26,6 +26,24 @@ After import, project list shows whether concepts are indexed for the selected ![](_static/img/concepts-imported-status.png) +## Document text-search index + +Clinical Notes **Text** search (annotate entities page) uses a separate Solr +collection from concept lookup. New datasets are indexed automatically after +upload when `DOCUMENT_SEARCH_INDEX_ENABLED=1`. + +To index datasets uploaded before text search was available: + +1. Open `/admin`. +2. Go to **Datasets**. +3. Select one or more datasets. +4. Run **Index selected datasets for text search**. + +Until indexing finishes (or if Solr is down), text search falls back to a +database match. See [Maintenance](maintenance.md) for the collection name and +ops checks. + + ## Downloading annotations From Django admin (`/admin` -> **Project annotate entities**), use bulk actions diff --git a/medcat-trainer/docs/index.md b/medcat-trainer/docs/index.md index 66f72b921..960e83d4f 100644 --- a/medcat-trainer/docs/index.md +++ b/medcat-trainer/docs/index.md @@ -20,8 +20,9 @@ review-only workflows (collect annotations without changing the model). - Terminate - Irrelevant - Collect optional **meta annotations** and **relation annotations**. -- Use **Project Groups** for multi-annotator setups. -- Run **metrics reports** across one or more compatible projects. +- Use **Project Groups** for multi-annotator setups (from `/project-admin` or the Python client). +- Run **metrics reports** across one or more compatible projects, including Concept Summary filtering of low-F1 concepts. +- Search clinical notes by **name or full text** in the annotator Clinical Notes panel. - Explore concept hierarchies and export concept filters. ## Typical workflow diff --git a/medcat-trainer/docs/installation.md b/medcat-trainer/docs/installation.md index 93a7e8c46..fa9baa1b3 100644 --- a/medcat-trainer/docs/installation.md +++ b/medcat-trainer/docs/installation.md @@ -123,6 +123,8 @@ Host-level Compose variables (for example port overrides) can be set by copying | `CSRF_TRUSTED_ORIGINS` | Allowed origins to access the admin panel. Mandatory to set if you expose the app over a different URL or port (default `'http://127.0.0.1:8001', 'http://localhost:8001'`). | | `MCTRAINER_PORT` | Host port for the web UI/API (default `8001`). | | `SOLR_PORT` | Host port for Solr admin (default `8983`). | +| `DOCUMENT_SEARCH_COLLECTION` | SolrCloud collection used for clinical-note full-text search (default `medcattrainer_documents`). | +| `DOCUMENT_SEARCH_INDEX_ENABLED` | Index uploaded datasets for text search (`1`/`0`, default `1`). | | `MEDCAT_CONFIG_FILE` | MedCAT config file path inside the container. | | `LOAD_EXAMPLES` | Load example model pack + dataset + project on startup (`1`/`0`). | | `PROVISIONING_CONFIG_PATH` | File path of a yaml defining projects to create on startup | diff --git a/medcat-trainer/docs/maintenance.md b/medcat-trainer/docs/maintenance.md index 672df314c..cc9836116 100644 --- a/medcat-trainer/docs/maintenance.md +++ b/medcat-trainer/docs/maintenance.md @@ -27,10 +27,13 @@ Database migrations are applied automatically on container startup. - Concept search availability: verify Solr container and project concept import status. - Document text search: each uploaded dataset is indexed into the Solr collection - `DOCUMENT_SEARCH_COLLECTION` (default `medcattrainer_documents`) by a background task. - Datasets uploaded before this was available can be indexed via the Django admin - Datasets action "Index selected datasets for text search". Until a dataset is fully - indexed, or if Solr is unavailable, text search falls back to a slower database match. + `DOCUMENT_SEARCH_COLLECTION` (default `medcattrainer_documents`) by a background task + when `DOCUMENT_SEARCH_INDEX_ENABLED=1`. Datasets uploaded before this was available + can be indexed via the Django admin Datasets action + "Index selected datasets for text search" (see [Django Admin](django_admin.md)). + Until a dataset is fully indexed, or if Solr is unavailable, text search falls back + to a slower database match. Annotators use **Name** / **Text** modes in the Clinical + Notes panel — see [Annotator Guide](annotator_guide.md#searching-clinical-notes). ## Backup and restore (SQLite deployments) diff --git a/medcat-trainer/docs/project_admin.md b/medcat-trainer/docs/project_admin.md index 9deff1349..6d9bf3fa5 100644 --- a/medcat-trainer/docs/project_admin.md +++ b/medcat-trainer/docs/project_admin.md @@ -19,9 +19,10 @@ The following user guide covers. `/project-admin` ## Project Administration 1. Open `/project-admin`. -2. Go to the **Projects** tab and select **+ Create New Project**. To edit an existing project select the project row. -3. Complete the forms for your project. Specifics of each field are discussed below. -4. Save. +2. Go to the **Projects** tab and select **+ Create New Project**. To create several annotator projects that share one configuration, switch the create form to **Project Group** instead — see [Project Groups](project_group_admin.md). +3. To edit an existing project, select the project row. +4. Complete the forms for your project. Specifics of each field are discussed below. +5. Save. ### Basic Settings @@ -126,3 +127,30 @@ From the home **Projects** table: - Save current model state. - Select compatible projects and submit a metrics report. +## Metrics reports + +Metrics reports compare model predictions against validated annotations across one or more compatible projects (same underlying model configuration). + +### Submitting a report + +1. On the home screen, select one or more compatible projects in the **Projects** table. +2. Use the metrics action / **Submit metrics report** control. +3. Open `/metrics-reports/` to watch job status, open completed reports, or delete failed/finished jobs. + +Only projects that share a compatible model setup should be combined. Reports with meta-annotation data also expose a **Meta Annotations** tab — see [Meta Annotations](meta_annotations.md). + +### Concept Summary filtering + +Open a completed report and go to the **Concept Summary** tab. Aggregate scores (macro/micro F1, precision, recall) and the concept table reflect only the concepts currently included. + +| Control | Purpose | +|---------|---------| +| **Low F1 below** | Threshold (default `0.7`) that marks low-performing concepts. | +| **Exclude N low performers** | Removes every included concept whose F1 is below the threshold. | +| Per-row **Exclude** | Removes a single concept from the scores. | +| Excluded tray / **Restore** | Lists excluded concepts; restore one concept or **Restore all**. | + +While concepts are excluded, each summary metric shows the filtered value plus a delta against the full-set baseline. Use this to estimate how overall performance would look if low-performing concepts were removed from evaluation (for example before deciding where to focus further annotation). + +Click TP / FP / FN counts in a concept row to open example dialogs for those predictions. + diff --git a/medcat-trainer/docs/project_group_admin.md b/medcat-trainer/docs/project_group_admin.md index d15e88049..775a2b817 100644 --- a/medcat-trainer/docs/project_group_admin.md +++ b/medcat-trainer/docs/project_group_admin.md @@ -15,13 +15,22 @@ Use a group when you need: ## Creating a Project Group -Create groups from Django admin (`/admin` -> **Project groups**). +Prefer **Project Admin** (`/project-admin`) for day-to-day group creation. -The key option is **Create Associated Projects**. +1. Open `/project-admin` and go to the **Projects** tab. +2. Select **+ Create New Project**, then switch the form type to **Project Group**. +3. Configure shared settings (dataset, model, concept filters, tasks, and so on). +4. Under **Group Members**: + - **Administrators** can see and manage every project in the group (you are added automatically). + - **Annotators** each receive their own annotation project when associated projects are created. +5. Leave **Create an annotation project for each annotator** checked unless you only want the group shell. +6. Save. -### `create_associated_projects = true` +You can still create or edit groups in Django admin (`/admin` -> **Project groups**) when you need low-level control. -On initial save, MedCATtrainer automatically creates one +### `create_associated_projects = true` (default) + +On save, MedCATtrainer automatically creates one `ProjectAnnotateEntities` per selected annotator. Naming pattern: @@ -52,10 +61,14 @@ home page and inspect projects within each group. Regular annotators typically work with individual projects and may not need the group view. +## Automating with the Python client + +`mctclient` can create and list project groups for provisioning scripts and CI. +See [Python Client](client.md). + ## Best practices - For inter-annotator agreement studies, disable `train_model_on_submit` in all group projects. - Keep naming conventions consistent for easy report comparison. - Prefer one shared configuration template per annotation campaign. -