MAINT Prefer public package imports and enforce export boundaries by romanlutz · Pull Request #3848 · microsoft/PyRIT

Roman Lutz (romanlutz) · 2026-11-03T06:12:32Z

Description

Imports through implementation filenames unnecessarily couple callers to the package layout. Prefer existing public exports, for example from pyrit.analytics import analyze_results instead of importing from pyrit.analytics.result_analysis.

Shorten imports across library code, tests, and paired notebook examples without adding exports or flattening distinct executor APIs. Existing import paths remain available; this does not remove APIs.
Add an export-aware check-public-imports pre-commit hook. It reads annotated or unannotated _LAZY_EXPORTS statically, follows chained re-exports, and reports the preferred import without loading PyRIT or optional dependencies.
Check changed files on local commits and all files in CI. After changing exports, run uv run pre-commit run check-public-imports --all-files to check unchanged consumers too. Direct invocation without filenames also retains a full tracked-file scan.
Keep tests in scope so public API tests do not depend on implementation filenames. Keep independent notebook checking because synchronization with paired Python files is not enforced.
Preserve internal imports within the package owning the export, package initializers, immutable Alembic revisions, and helpers without public exports. Update affected mock lookup paths and Ruff's runtime-evaluated base-class configuration.
Ban relative imports with Ruff TID252 so sibling-relative imports cannot bypass the checker.

Ruff's TID251 supports fixed banned API paths, but does not infer public re-exports or selectively permit internal sibling imports. The hook avoids maintaining a second API inventory. It checks absolute from ... import ... statements, including deferred and TYPE_CHECKING imports, in Python files and notebook code cells. Plain module imports are not checked.

Patch lazily exported names where the consumer looks them up. For deferred public imports, patch the public package attribute, not its implementation module, to avoid cached mocks or bypassed patches. Internal consumers still need implementation-side patches; module-level imports should be patched at the consumer's bound name.

Tests and Documentation

Updated contributor and repository style guidance, including lazy-export patching examples. Added 31 checker tests covering chained exports, aliases, internal imports, separate executor packages, notebook syntax and independent checking, both export-assignment forms, exclusions, and error reporting. Review fixes and the first main merge are pushed in 34cc83b69cd14d6603d250d26210c3b7d5e334a8.

Validation after review fixes and merging main at 4b0383ce9181665de57f6517802172d285df96f1:

uv run --frozen --extra all pytest -n 4 --dist=loadfile tests\unit -q --tb=short --disable-warnings: 34,673 passed, 47 skipped, 4 xfailed; 135 subtests passed. No failures or collection errors. This is the Makefile-equivalent command because make is unavailable on this Windows host. An earlier run exposed one merge-related import collection error, which was fixed before this complete rerun.
uv run --no-sync pytest -q tests\unit\build_scripts\test_check_public_imports.py tests\unit\output\scorer\test_pretty.py --tb=short: 51 passed. Both affected scorer tests also passed in fresh processes with the lazy export cached and initially uncached, with no mock leak after execution.
uv run --frozen --extra all pytest -q tests\unit\auth\test_azure_auth.py --tb=short: 52 passed after correcting the non-exported helper import.
uv run --no-sync python -m build_scripts.check_public_imports: passed.
uv run --no-sync ty check pyrit tests\unit --output-format concise: passed.
uv run --frozen --extra all pre-commit run --all-files: all hooks passed, including Ruff, notebook lint, memory migrations, typing, and the public import check.
uv run --frozen --extra all pre-commit run memory-migrations-check --all-files: passed again immediately before committing. The installed Git hook launches this system-language check with mismatched Python runtimes on this host, so that hook alone was skipped for the commit after this direct successful run. Other applicable commit hooks passed.

Main advanced by two commits during validation. Merged those changes through 19b615bdb58f in bf853af1b288f88bead6e38ddffdf7458e45f0ab, resolving one import-only conflict while retaining the new exception helper. Post-merge validation:

uv run --frozen --extra all pytest -n 4 --dist=loadfile tests\unit\backend\test_runtime_lifecycle.py tests\unit\backend\test_scenario_run_service.py tests\unit\datasets\test_anthropic_model_written_evals_dataset.py tests\unit\exceptions\test_exceptions_helpers.py tests\unit\scenario\core -q --tb=short --disable-warnings: 1,284 passed.
uv run --no-sync python -m build_scripts.check_public_imports: passed full tracked-file scan.
uv run --no-sync ty check pyrit tests\unit --output-format concise: passed.
uv run --frozen --extra all pre-commit run: all applicable merge-conflict hooks passed. All applicable commit hooks also passed.

The full unit suite was not rerun after this second small main merge. Its frontend changes came directly from main and were not locally tested.

AST comparison against the first merged main confirmed that all 244 changed library files differ only in imports. Ruff's relative-import rule passes on tracked Python files and rejects a synthetic sibling-relative import.

Compared code cells using Jupytext for all 17 changed notebook pairs: synchronized, with notebook outputs and metadata preserved. Notebooks were not executed against live services. Integration and end-to-end tests were not run.

Shorten implementation-module imports to existing public exports across the library, tests, and paired notebook examples. Add an export-aware pre-commit guard, preserve internal sibling imports and frozen migrations, and update lookup-site test patches. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

The image-display tests still patch the original helper instead of the cached pyrit.common export, which can bypass image-rendering coverage or leak a mock between tests.

1 open issue

Patch cached is_in_ipython_session export in image-display tests · New

What changed in this PR

This PR reduces coupling to PyRIT’s internal layout by replacing implementation-path imports with existing public package exports and adding export-aware import enforcement.

Changes:

Standardizes imports across library code, tests, scripts, and documentation examples.
Adds a static public-import checker while preserving internal helpers and distinct executor APIs.
Updates notebook pairs, mock lookup paths, import-boundary allowances, and Ruff configuration.

This pull request requires changes. Two moderate issues remain:

pyrit/output/conversation/pretty.py:366: Image-display tests still patch the implementation-module notebook detector. Update them to patch pyrit.common.is_in_ipython_session to avoid cached-export interference and preserve image-rendering coverage.
pyrit/output/scorer/pretty.py:396: Two rendering/color tests still patch the implementation-module metrics lookup. Update those decorators to patch pyrit.score.find_objective_metrics_by_eval_hash; otherwise, cached exports bypass their configured mocks and silently weaken coverage.

File	Description
`tests/unit/test_show_versions.py`	Uses public version-reporting export.
`tests/unit/setup/test_tools_initializer.py`	Uses public registry export.
`tests/unit/setup/test_targets_initializer.py`	Uses public target initializer export.
`tests/unit/setup/test_refresh_datasets.py`	Uses public refresh initializer export.
`tests/unit/setup/test_preload_scenario_metadata.py`	Uses public metadata initializer export.
`tests/unit/setup/test_initialization.py`	Consolidates public common imports.
`tests/unit/setup/test_configuration_loader.py`	Uses public setup exports.
`tests/unit/score/test_true_false_inverter.py`	Consolidates public memory imports.
`tests/unit/score/test_true_false_composite_scorer.py`	Uses public memory export.
`tests/unit/score/test_system_prompt_extraction_scorer.py`	Consolidates public memory imports.
`tests/unit/score/test_substring.py`	Consolidates public memory imports.
`tests/unit/score/test_shieldgemma_scorer.py`	Uses public memory interface export.
`tests/unit/score/test_self_ask_refusal.py`	Uses public exception and memory exports.
`tests/unit/score/test_self_ask_likert.py`	Uses public exception export.
`tests/unit/score/test_self_ask_category.py`	Uses public exception and memory exports.
`tests/unit/score/test_scorer_metrics.py`	Uses public metric-query exports.
`tests/unit/score/test_response_scoring_expectation.py`	Uses public aggregator type.
`tests/unit/score/test_response_handler.py`	Uses public response-handler exports.
`tests/unit/score/test_metrics_type.py`	Uses public metrics enums.
`tests/unit/score/test_message_scorable_resolver.py`	Uses public resolver export.
`tests/unit/score/test_judgment_observation.py`	Uses public prompt-memory model.
`tests/unit/score/test_insecure_code_scorer.py`	Uses public exception export.
`tests/unit/score/test_decoding_scorer.py`	Consolidates public memory imports.
`tests/unit/score/test_callable_response_handler.py`	Uses public exception export.
`tests/unit/score/test_batch_scorer.py`	Uses public compound-attack exports.
`tests/unit/score/regex/test_xxe_output_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_xss_output_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_static_prompt_injection_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_ssti_output_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_sql_injection_output_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_shell_command_output_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_path_traversal_output_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_open_redirect_output_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_nerve_agent_keyword_scorer.py`	Uses public regex scorer export.
`tests/unit/score/regex/test_meth_keyword_scorer.py`	Uses public regex scorer export.
`tests/unit/score/regex/test_markdown_injection.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_ldap_injection_output_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_fentanyl_keyword_scorer.py`	Uses public regex scorer export.
`tests/unit/score/regex/test_escaped_ansi_output_scorer.py`	Consolidates public memory imports.
`tests/unit/score/regex/test_divergence_scorer.py`	Uses public aggregator type.
`tests/unit/score/regex/test_anthrax_keyword_scorer.py`	Uses public regex scorer export.
`tests/unit/score/regex/test_ansi_escape_output_scorer.py`	Consolidates public memory imports.
`tests/unit/scenario/test_scenario_e2e_contract.py`	Uses AIRT package export.
`tests/unit/scenario/scenarios/adaptive/test_technique_selector.py`	Uses adaptive selector exports.
`tests/unit/scenario/scenarios/adaptive/test_selector_scope.py`	Uses adaptive scope export.
`tests/unit/scenario/scenarios/adaptive/test_epsilon_greedy.py`	Uses analytics and adaptive exports.
`tests/unit/scenario/garak/test_figstep.py`	Uses public dataset configuration.
`tests/unit/scenario/garak/conftest.py`	Uses public dataset provider.
`tests/unit/scenario/core/test_scenario_partial_results.py`	Uses public executor-result export.
`tests/unit/scenario/core/test_scenario_context.py`	Uses public dataset configuration.
`tests/unit/scenario/core/test_dataset_isolation.py`	Consolidates dataset configuration imports.
`tests/unit/scenario/core/test_attack_technique.py`	Uses public attack technique export.
`tests/unit/scenario/core/test_attack_constructor_compatibility.py`	Uses public attack scoring configuration.
`tests/unit/scenario/conftest.py`	Uses public dataset configuration.
`tests/unit/registry/test_tag_query.py`	Uses public tag-query export.
`tests/unit/registry/test_scenario_suite_registry.py`	Uses public scenario suite models.
`tests/unit/registry/test_scenario_registry.py`	Uses public scenario registry export.
`tests/unit/registry/test_registry.py`	Uses public registry classes and metadata.
`tests/unit/registry/test_registry_metadata.py`	Uses public registry metadata.
`tests/unit/registry/test_instance_registry.py`	Consolidates public registry imports.
`tests/unit/registry/test_attack_technique_construction.py`	Uses public required-value sentinel.
`tests/unit/registry/test_attack_registry.py`	Uses public component-type export.
`tests/unit/prompt_target/test_round_robin_target.py`	Uses public capability-policy exports.
`tests/unit/prompt_target/target/test_openai_target_auth.py`	Uses public authentication and target exports.
`tests/unit/prompt_target/target/test_http_target.py`	Uses public HTTP target and callback exports.
`tests/unit/prompt_target/target/test_http_target_parsing.py`	Uses public HTTP target and callback exports.
`tests/unit/prompt_target/target/test_http_api_target.py`	Uses public HTTPX target export.
`tests/unit/prompt_target/target/test_azure_openai_completion_target.py`	Uses public central-memory export.
`tests/unit/prompt_normalizer/test_converter_configuration.py`	Uses public converter configuration.
`tests/unit/output/test_sink.py`	Uses public sink exports.
`tests/unit/models/test_yaml_seed_loader.py`	Uses public seed models; retains loader helpers.
`tests/unit/models/test_seed_simulated_conversation.py`	Uses public seed models.
`tests/unit/models/test_seed_prompt.py`	Uses public seed prompt export.
`tests/unit/models/test_seed_objective.py`	Uses public seed objective export.
`tests/unit/models/test_seed_group.py`	Consolidates public seed-group imports.
`tests/unit/models/test_seed_dataset.py`	Consolidates public seed-model imports.
`tests/unit/models/test_score.py`	Uses public unvalidated-score export.
`tests/unit/models/test_scenario_result.py`	Consolidates result and retry imports.
`tests/unit/models/test_scenario_hierarchy_models.py`	Uses catalog summary exports.
`tests/unit/models/test_scenario_catalog.py`	Uses catalog run-request export.
`tests/unit/models/test_retry_event.py`	Uses public retry-event export.
`tests/unit/models/test_question_answering.py`	Uses public question-answering models.
`tests/unit/models/test_operation.py`	Uses public operation models; retains helper.
`tests/unit/models/test_message_piece.py`	Uses public checksum helper.
`tests/unit/models/test_import_boundary.py`	Aligns allowlist with public import.
`tests/unit/models/test_harm_definition.py`	Uses public harm-definition exports.
`tests/unit/models/test_finding.py`	Uses public finding models.
`tests/unit/models/test_conversation_stats.py`	Uses public conversation statistics.
`tests/unit/models/test_condition.py`	Uses public tool-call export with alias.
`tests/unit/models/test_attack_technique_seed_group.py`	Consolidates public seed-group imports.
`tests/unit/models/test_attack_seed_group.py`	Consolidates public seed-model imports.
`tests/unit/models/test_analytics.py`	Routes analytics aliases through public export.
`tests/unit/models/identifiers/test_identifier_projection.py`	Uses public identifier exports.
`tests/unit/models/identifiers/test_identifier_filters.py`	Uses public identifier filters.
`tests/unit/models/identifiers/test_class_name_utils.py`	Uses public identifier utilities.
`tests/unit/mocks.py`	Uses public scenario technique export.
`tests/unit/message_normalizer/test_history_squash_normalizer.py`	Uses public chat-role type.
`tests/unit/message_normalizer/test_generic_system_squash_normalizer.py`	Uses public chat-role type.
`tests/unit/message_normalizer/test_conversation_context_normalizer.py`	Uses public message literal exports.
`tests/unit/message_normalizer/test_chat_normalizer_tokenizer.py`	Uses public chat-role type.
`tests/unit/message_normalizer/test_chat_message_normalizer.py`	Consolidates public message-type imports.
`tests/unit/memory/test_seed_origin_migration.py`	Uses public seed-entry export.
`tests/unit/memory/test_operations_and_findings_migration.py`	Uses public attack-result entry.
`tests/unit/memory/test_identifier_filters.py`	Consolidates public memory imports.
`tests/unit/memory/test_conversation_branching.py`	Uses public attack-result entry.
`tests/unit/memory/test_activity_operation_ids.py`	Uses public attack-result entry.
`tests/unit/memory/storage/test_storage.py`	Uses public storage exports.
`tests/unit/memory/storage/test_data_url_converter.py`	Uses storage-package conversion helper.
`tests/unit/memory/memory_interface/test_interface_seed_prompts.py`	Uses public seed-entry export.
`tests/unit/memory/memory_interface/test_interface_seed_examples.py`	Uses public seed-entry export.
`tests/unit/memory/memory_interface/test_interface_core.py`	Uses public embedding and prompt-memory entries.
`tests/unit/memory/memory_interface/test_batching_scale.py`	Uses public prompt-memory entry.
`tests/unit/executor/promptgen/test_anecdoctor.py`	Uses executor and generator package exports.
`tests/unit/executor/promptgen/gcg/test_config.py`	Uses GCG configuration exports.
`tests/unit/executor/promptgen/fuzzer/test_fuzzer_converter.py`	Uses public exception export.
`tests/unit/executor/core/test_strategy.py`	Uses executor-core strategy exports.
`tests/unit/executor/core/test_config.py`	Uses core converter configuration.
`tests/unit/executor/benchmark/test_question_answering.py`	Uses benchmark package exports.
`tests/unit/executor/benchmark/test_fairness_bias.py`	Uses benchmark package exports.
`tests/unit/executor/attack/multi_turn/test_pair.py`	Uses public TAP context export.
`tests/unit/executor/attack/core/test_attack_executor.py`	Uses public executor-result export.
`tests/unit/executor/attack/component/test_prepended_conversation_config.py`	Uses public prepended-conversation configuration.
`tests/unit/exceptions/test_json_extraction_ambiguity.py`	Uses public Markdown cleanup helper.
`tests/unit/exceptions/test_exceptions_helpers.py`	Uses public Markdown cleanup helper.
`tests/unit/docs/test_converter_documentation.py`	Uses public converter modality utility.
`tests/unit/docs/test_attack_documentation.py`	Uses public attack strategy export.
`tests/unit/datasets/test_wildguardmix_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_visual_leak_bench_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_transphobia_awareness_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_toxic_chat_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_strong_reject_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_sosbench_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_sorry_bench_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_siuo_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_simple_safety_tests_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_sgxstest_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_semguard_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_seed_dataset_metadata.py`	Uses public dataset metadata.
`tests/unit/datasets/test_salad_bench_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_remote_dataset_origin.py`	Consolidates remote dataset imports.
`tests/unit/datasets/test_remote_dataset_loader.py`	Uses provider and loader package exports.
`tests/unit/datasets/test_red_team_social_bias_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_promptintel_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_pku_safe_rlhf_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_or_bench_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_odin_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_multilingual_vulnerability_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_msts_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_mossbench_dataset.py`	Uses dataset and memory package exports.
`tests/unit/datasets/test_moral_integrity_corpus_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_mm_safetybench_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_mlcommons_ailuminate_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_medsafetybench_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_mask_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_local_dataset_loader.py`	Uses local dataset loader export.
`tests/unit/datasets/test_jbb_behaviors_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_hixstest_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_harmful_qa_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_harmeval_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_harmbench_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_garak_system_prompt_dataset.py`	Uses remote dataset loader exports.
`tests/unit/datasets/test_garak_propile_dataset.py`	Uses local dataset loader export.
`tests/unit/datasets/test_garak_prompt_inject_dataset.py`	Uses local dataset loader export.
`tests/unit/datasets/test_garak_package_hallucination_dataset.py`	Uses remote dataset loader exports.
`tests/unit/datasets/test_garak_exploitation_dataset.py`	Uses local dataset loader export.
`tests/unit/datasets/test_garak_audio_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_garak_api_key_dataset.py`	Uses local dataset loader export.
`tests/unit/datasets/test_forbidden_questions_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_equitymedqa_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_darkbench_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_dangerous_qa_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_cyberseceval_prompt_injection_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_cyberseceval_mitre_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_coconot_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_ccp_sensitive_prompts_dataset.py`	Uses loader export; retains private helper.
`tests/unit/datasets/test_cbt_bench_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_categorical_harmful_qa_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_beaver_tails_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_babelscape_alert_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_aya_redteaming_dataset.py`	Uses remote dataset loader export.
`tests/unit/datasets/test_agent_threat_rules_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_aegis_ai_content_safety_dataset.py`	Uses remote dataset package exports.
`tests/unit/datasets/test_0din_nday_datasets.py`	Uses public seed prompt export.
`tests/unit/converter/test_zalgo_converter.py`	Uses public word-selection exports.
`tests/unit/converter/test_word_level_converter.py`	Uses public selection strategy exports.
`tests/unit/converter/test_variation_converter.py`	Uses public exception export.
`tests/unit/converter/test_unicode_confusable_converter.py`	Uses public converter export.
`tests/unit/converter/test_token_smuggler_converter.py`	Uses public token-smuggling exports.
`tests/unit/converter/test_string_join_converter.py`	Consolidates converter and selection imports.
`tests/unit/converter/test_random_translation_converter.py`	Uses public prompt-target export.
`tests/unit/converter/test_puzzled_puzzle_builders.py`	Uses public puzzle-type export.
`tests/unit/converter/test_persuasion_converter.py`	Uses public exception export.
`tests/unit/converter/test_nato_converter.py`	Uses public word-selection export.
`tests/unit/converter/test_math_prompt_converter.py`	Uses public math prompt converter.
`tests/unit/converter/test_math_obfuscation_converter.py`	Uses public converter export.
`tests/unit/converter/test_leetspeak_converter.py`	Uses public converter export.
`tests/unit/converter/test_image_url_downloader.py`	Uses public image compression converter.
`tests/unit/converter/test_image_prompt_style_converter.py`	Uses public prompt-target export.
`tests/unit/converter/test_generic_llm_converter.py`	Uses public prompt-target export.
`tests/unit/converter/test_emoji_converter.py`	Uses public word-selection export.
`tests/unit/converter/test_diacritics_converter.py`	Uses public diacritic converter.
`tests/unit/converter/test_colloquial_wordswap_converter.py`	Uses public converter export.
`tests/unit/converter/test_char_swap_generator_converter.py`	Consolidates converter and selection imports.
`tests/unit/converter/test_binary_converter.py`	Uses public word-selection export.
`tests/unit/converter/test_audio_white_noise_converter.py`	Uses public audio converter export.
`tests/unit/converter/test_audio_volume_converter.py`	Uses public audio converter export.
`tests/unit/converter/test_audio_speed_converter.py`	Uses public audio converter export.
`tests/unit/converter/test_audio_frequency_converter.py`	Uses public audio converter export.
`tests/unit/converter/test_audio_echo_converter.py`	Uses public audio converter export.
`tests/unit/converter/test_ansi_attack_converter.py`	Uses public ANSI converter export.
`tests/unit/common/test_yaml_loadable.py`	Uses public YAML-loadable export.
`tests/unit/common/test_utils.py`	Uses public list-combination helper.
`tests/unit/common/test_singleton.py`	Uses public singleton export.
`tests/unit/common/test_pyrit_default_value.py`	Uses public default-value exports.
`tests/unit/common/test_mime_type.py`	Uses public MIME utility.
`tests/unit/common/test_helper_functions.py`	Uses public common utilities.
`tests/unit/common/test_deprecation.py`	Uses public deprecation helper.
`tests/unit/common/test_brick_contract.py`	Uses public constructor-contract exports.
`tests/unit/cli/test_sources.py`	Uses public backend score view.
`tests/unit/cli/test_scenario_report_parity.py`	Uses public attack service.
`tests/unit/backend/test_target_types_concurrency.py`	Uses public target service.
`tests/unit/backend/test_target_service.py`	Uses public backend models and services.
`tests/unit/backend/test_security_headers.py`	Uses middleware package export.
`tests/unit/backend/test_scorer_routes.py`	Uses public scorer export.
`tests/unit/backend/test_response_contracts.py`	Uses backend model package exports.
`tests/unit/backend/test_request_size.py`	Uses public error-handler registration.
`tests/unit/backend/test_request_id.py`	Uses middleware request-ID export.
`tests/unit/backend/test_operations.py`	Uses public memory entries.
`tests/unit/backend/test_message_send_routes.py`	Uses public attack service.
`tests/unit/backend/test_finding_export.py`	Uses public backend score view.
`tests/unit/backend/test_error_handlers.py`	Uses public error-handler registration.
`tests/unit/backend/test_dataset_service.py`	Uses public dataset service exports.
`tests/unit/backend/test_compatibility.py`	Uses public configuration loader.
`tests/unit/auth/test_openai_auth.py`	Uses public authentication resolver.
`tests/unit/auth/test_manual_copilot_authenticator.py`	Uses public authenticator export.
`tests/unit/auth/test_copilot_authenticator.py`	Uses public authenticator export.
`tests/unit/auth/test_authenticator.py`	Uses public authenticator base.
`tests/unit/analytics/test_text_matching.py`	Uses public text-matching exports.
`tests/unit/analytics/test_scenario_statistics.py`	Uses public scenario statistics export.
`tests/unit/analytics/test_scenario_statistics_parity.py`	Uses public scenario-run service.
`tests/partner_integration/conftest.py`	Consolidates public memory imports.
`tests/partner_integration/azure_ai_evaluation/test_import_smoke.py`	Uses public true/false scorer.
`tests/integration/targets/test_targets_and_secrets.py`	Uses public SQLite memory.
`tests/integration/memory/test_seed_examples_azure_sql_integration.py`	Uses public seed-entry export.
`tests/integration/datasets/test_load_default_datasets_integration.py`	Uses public initializer exports.
`tests/integration/conftest.py`	Consolidates public memory imports.
`tests/end_to_end/browser_initializer.py`	Uses public attack-result entry.
`test_support/fixtures.py`	Uses public singleton export.
`pyrit/setup/initializers/tools.py`	Uses public technique factory.
`pyrit/setup/initializers/techniques/image.py`	Uses public technique factory.
`pyrit/setup/initializers/techniques/core.py`	Uses public technique configuration exports.
`pyrit/setup/initializers/scorers.py`	Uses public parameter model.
`pyrit/setup/initializers/refresh_datasets.py`	Uses public parameter model.
`pyrit/setup/initializers/load_default_datasets.py`	Uses public parameter model.
`pyrit/score/true_false/decoding_scorer.py`	Uses public central-memory export.
`pyrit/score/text_matching.py`	Uses public text-matcher models.
`pyrit/score/message_scorer.py`	Uses public deprecation helper.
`pyrit/score/message_scorable_resolver.py`	Uses public deprecation helper.
`pyrit/scenario/scenarios/airt/multilingual.py`	Uses public technique registry.
`pyrit/scenario/core/scenario_technique.py`	Uses public deprecation helper.
`pyrit/scenario/core/scenario_target_defaults.py`	Uses public capability-name export.
`pyrit/scenario/core/dataset_overrides.py`	Uses public dataset request models.
`pyrit/scenario/core/_technique_resolution.py`	Uses public technique registry.
`pyrit/scenario/core/_attack_constructor_compatibility.py`	Uses public attack scoring configuration.
`pyrit/registry/scenario_suite_storage.py`	Uses public scenario suite models.
`pyrit/registry/scenario_preset_storage.py`	Uses public scenario preset models.
`pyrit/registry/registry.py`	Consolidates public model-type imports.
`pyrit/registry/registry_metadata.py`	Uses public parameter type.
`pyrit/registry/file_document_storage.py`	Uses public registry-name validator.
`pyrit/registry/components/target_registry.py`	Uses public target identifier.
`pyrit/registry/components/initializer_registry.py`	Consolidates public model-type imports.
`pyrit/registry/components/converter_registry.py`	Uses public component and converter identifiers.
`pyrit/registry/components/attack_registry.py`	Uses public attack identifier.
`pyrit/prompt_target/websocket_target.py`	Uses public deprecation helper.
`pyrit/prompt_target/prompt_shield_target.py`	Consolidates common package imports.
`pyrit/prompt_target/openai/openai_target.py`	Uses public exception exports.
`pyrit/prompt_target/openai/openai_realtime_target.py`	Consolidates public common imports.
`pyrit/prompt_target/openai/openai_error_handling.py`	Uses public content-filter markers.
`pyrit/prompt_target/openai/openai_completion_target.py`	Uses public retry decorator.
`pyrit/prompt_target/http_target/httpx_api_target.py`	Uses public deprecation helper.
`pyrit/prompt_target/common/target_capabilities.py`	Uses public capability models.
`pyrit/prompt_target/azure_ml_chat_target.py`	Consolidates common package imports.
`pyrit/prompt_target/a2a_target.py`	Uses public content-filter markers.
`pyrit/prompt_normalizer/normalizer_request.py`	Uses public capability-name export.
`pyrit/output/sink.py`	Uses public notebook-detection helper.
`pyrit/output/scorer/pretty.py`	Uses public objective-metrics lookup.
`pyrit/output/scorer/json.py`	Uses public objective-metrics lookup.
`pyrit/output/conversation/pretty.py`	Uses public notebook-detection helper.
`pyrit/output/_derivation.py`	Uses public scenario statistics export.
`pyrit/models/seeds/yaml_seed_loader.py`	Uses public path-validation helper.
`pyrit/models/seeds/seed_simulated_conversation.py`	Uses public deprecation helper.
`pyrit/models/seeds/seed_dataset.py`	Uses public list-combination helper.
`pyrit/models/scenario_suite_execution.py`	Uses catalog run-summary export.
`pyrit/models/scenario_hierarchy.py`	Uses catalog error and retry summaries.
`pyrit/models/results/scenario_result.py`	Uses public deprecation helper.
`pyrit/models/results/attack_result.py`	Uses public deprecation helper.
`pyrit/models/parameter.py`	Uses public required-value sentinel.
`pyrit/models/identifiers/evaluation_identifier.py`	Uses public attack strategy type.
`pyrit/message_normalizer/chat_message_normalizer.py`	Uses public chat-role type.
`pyrit/memory/storage/serializers.py`	Consolidates public model-type imports.
`pyrit/memory/sqlite_memory.py`	Uses public singleton export.
`pyrit/memory/memory_models.py`	Uses public retry-event export.
`pyrit/executor/workflow/xpia.py`	Uses public utilities; retains internal helper.
`pyrit/executor/workflow/core/workflow_strategy.py`	Uses executor-core strategy exports.
`pyrit/executor/promptgen/gcg/generator.py`	Uses public dictionary-combination helper.
`pyrit/executor/promptgen/fuzzer/fuzzer_similar_converter.py`	Uses public defaults decorator.
`pyrit/executor/promptgen/fuzzer/fuzzer_shorten_converter.py`	Uses public defaults decorator.
`pyrit/executor/promptgen/fuzzer/fuzzer_rephrase_converter.py`	Uses public defaults decorator.
`pyrit/executor/promptgen/fuzzer/fuzzer_converter_base.py`	Uses public default-value exports.
`pyrit/executor/promptgen/core/prompt_generator_strategy.py`	Uses executor-core strategy exports.
`pyrit/executor/promptgen/assert_generator.py`	Uses public target requirements.
`pyrit/executor/attack/single_turn/skeleton_key.py`	Uses public default-value exports.
`pyrit/executor/attack/multi_turn/simulated_conversation.py`	Uses component-package manager export.
`pyrit/executor/attack/multi_turn/pair.py`	Uses public default-value exports.
`pyrit/exceptions/retry_collector.py`	Uses public retry-event export.
`pyrit/embedding/openai_text_embedding.py`	Uses public authentication-mode export.
`pyrit/datasets/seed_datasets/remote/xstest_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/vlguard_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/tdc23_redteaming_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/siuo_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/multilingual_vulnerability_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/msts_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/moral_integrity_corpus_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/mlcommons_ailuminate_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/librai_do_not_answer_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/figstep_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/equitymedqa_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/comic_jailbreak_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/coconot_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/ccp_sensitive_prompts_dataset.py`	Uses public seed-union type.
`pyrit/datasets/seed_datasets/remote/babelscape_alert_dataset.py`	Uses public seed-union type.
`pyrit/converter/variation_converter.py`	Uses public default-value exports.
`pyrit/converter/translation_converter.py`	Uses public default-value exports.
`pyrit/converter/toxic_sentence_generator_converter.py`	Uses public default-value exports.
`pyrit/converter/tone_converter.py`	Uses public default-value exports.
`pyrit/converter/token_smuggling/base.py`	Uses public prompt-data type.
`pyrit/converter/tense_converter.py`	Uses public default-value exports.
`pyrit/converter/task_framing_converter.py`	Uses public deprecation helper.
`pyrit/converter/scientific_translation_converter.py`	Uses public default-value exports.
`pyrit/converter/role_play_tool_call_converter.py`	Uses public default-value exports.
`pyrit/converter/random_translation_converter.py`	Uses public default-value exports.
`pyrit/converter/puzzled/puzzled_converter.py`	Uses public Markdown cleanup helper.
`pyrit/converter/persuasion_converter.py`	Uses public default-value exports.
`pyrit/converter/noise_converter.py`	Uses public default-value exports.
`pyrit/converter/math_prompt_converter.py`	Uses public default-value exports.
`pyrit/converter/malicious_question_generator_converter.py`	Uses public default-value exports.
`pyrit/converter/image_prompt_style_converter.py`	Uses public default-value exports.
`pyrit/converter/denylist_converter.py`	Uses public default-value exports.
`pyrit/converter/decomposition_converter.py`	Uses public default-value exports.
`pyrit/converter/binary_converter.py`	Uses public deprecation helper.
`pyrit/cli/pyrit_shell.py`	Uses public deprecation helper.
`pyrit/cli/pyrit_migrate.py`	Uses public configuration loader.
`pyrit/backend/services/technique_edit_service.py`	Uses public component-type export.
`pyrit/backend/services/runtime_lifecycle.py`	Uses public configuration loader.
`pyrit/backend/services/operation_service.py`	Uses public backend score view.
`pyrit/backend/services/media_persistence.py`	Uses public message-piece request type.
`pyrit/backend/services/finding_export.py`	Uses public backend score view.
`pyrit/backend/services/configuration_file_service.py`	Uses public configuration loader.
`pyrit/backend/routes/techniques.py`	Uses public problem-detail model.
`pyrit/backend/routes/scores.py`	Uses public backend response models.
`pyrit/backend/routes/analytics.py`	Uses public problem-detail model.
`pyrit/backend/models/scorers.py`	Uses catalog scorer-instance export.
`pyrit/backend/models/scenarios.py`	Uses public scenario catalog exports.
`pyrit/backend/models/scenario_suites.py`	Uses public suite and preset-issue models.
`pyrit/backend/models/scenario_presets.py`	Uses public scenario preset models.
`pyrit/backend/models/initializers.py`	Uses catalog initializer export.
`pyrit/backend/models/datasets.py`	Uses public dataset-settings model.
`pyrit/backend/middleware/error_handlers.py`	Uses public backend error models.
`pyrit/auth/openai_auth.py`	Consolidates public common imports.
`pyrit/analytics/conversation_analytics.py`	Uses public common and memory exports.
`pyproject.toml`	Recognizes public `StrategyResult` base in Ruff.
`frontend/e2e/fixtures/manual_send_backend.py`	Consolidates public setup imports.
`doc/scanner/1_pyrit_scan.py`	Demonstrates public OpenAI target import.
`doc/code/setup/pyrit_initializer.py`	Demonstrates public defaults helper import.
`doc/code/setup/pyrit_initializer.ipynb`	Mirrors defaults helper import update.
`doc/code/scenarios/0_attack_techniques.py`	Demonstrates public technique initializer import.
`doc/code/datasets/6_generated_datasets.py`	Uses scenario-core target helper export.
`doc/code/datasets/4_dataset_coding.py`	Demonstrates remote-package loader import.
`doc/code/datasets/4_dataset_coding.ipynb`	Mirrors remote-package loader import.
`doc/code/datasets/1_loading_datasets.py`	Demonstrates public setup imports.
`doc/code/datasets/1_loading_datasets.ipynb`	Mirrors public setup imports.
`doc/code/converters/3_image_converters.py`	Demonstrates public prompt-sending attack import.
`doc/code/converters/3_image_converters.ipynb`	Mirrors public attack import.
`build_scripts/export_adversarial_benchmark_result.py`	Uses public file-sink export.
`build_scripts/evaluate_scorers.py`	Uses public execution-context exports.

_{Review effort: Balanced}

Patch the public common export used by image rendering so cached lazy exports cannot bypass notebook-mode tests or retain mocked functions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Spencer Schoenberg (spencrr)

🤖 This review was generated with GHCP.

Checked out e0f8130 locally and reviewed the full diff (787 files). Net: one must-fix (two stale patch targets), the rest are size/perf suggestions on the new hook and nits. Labels: MUST FIX (blocking) · SUGGESTION (recommended, non-blocking) · NIT (take or leave); severity HIGH / MEDIUM / LOW.

Verified on this branch (the description notes the full suite was not rerun after the 18 mock fixes):

uv run --frozen --extra all pytest -n 8 --dist=loadfile tests/unit -q: 33,233 passed, 65 skipped, 0 failed (9m05s)
tests/unit/build_scripts/test_check_public_imports.py: 28 passed · ty check pyrit: clean · python -m build_scripts.check_public_imports: 0 findings
Zero __init__.py changes; zero non-import line changes under pyrit/; all 17 notebook .py/.ipynb pairs changed together; both runtime-evaluated-base-classes entries are still needed.
AST cross-check of the 43 deferred imports this PR made shallower against every string patch() target in tests/ → only the two in test_pretty.py (lines 302 / 353) remain stale.

Rebase soon: the branch is CONFLICTING and 67 commits behind main. Running the hook against current main shows 39 new violations in 17 files introduced since the merge-base (~0.6 per commit), so the fix-up grows the longer this waits.

PR description: once the test_pretty.py fix lands, please update the Tests and Documentation section — it currently states the full unit suite was not rerun after the 18 mock fixes. The clean run above can serve as interim evidence, but the description should reflect a run on the final head.

Follow-up, not for this PR (file is not in the diff, so noted here rather than inline): tests/unit/common/test_lazy_package_imports.py only verifies "every _LAZY_EXPORTS entry resolves to the declared module attribute" for the 29 hand-picked tuples in _LAZY_IMPORT_SPOT_CHECKS; test_lazy_package_runtime_contract_in_process checks dir() only, which get_lazy_dir satisfies without resolving anything. Now that the hook makes _LAZY_EXPORTS the enforced boundary for every caller, one parametrized in-process test over _non_exempt_public_initializers() × _LAZY_EXPORTS asserting getattr(package, name) is getattr(import_module(module), attr) (~1,500 cases, cheap) would cover all of them and let the spot-check table shrink to the "unrelated module not loaded" cases it is really there for.

Retain independent notebook and test enforcement, check changed files locally, support both lazy-export declaration forms, and document cache-safe patches. Fix remaining mock targets and enable Ruff's relative-import ban. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟢 Approval recommended

No active review issues were reported.

No open issues 🎉

1 resolved since last review

Patch cached is_in_ipython_session export in image-display tests

_{Review effort: Balanced}

Copilot AI balanced review requested due to automatic review settings November 3, 2026 06:12

Copilot started reviewing on behalf of Roman Lutz (romanlutz) November 3, 2026 06:13

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

Comment thread pyrit/output/conversation/pretty.py Resolved

Fix notebook detection mocks in image display tests

e0f8130

Patch the public common export used by image rendering so cached lazy exports cannot bypass notebook-mode tests or retain mocked functions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Spencer Schoenberg (spencrr) reviewed Nov 5, 2026

View reviewed changes

Roman Lutz (romanlutz) and others added 2 commits November 6, 2026 01:02

Merge latest main and preserve public scenario imports

bf853af

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Spencer Schoenberg (spencrr) requested a balanced review from Copilot November 6, 2026 19:30

Copilot started reviewing on behalf of Spencer Schoenberg (spencrr) November 6, 2026 19:30

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Spencer Schoenberg (spencrr) approved these changes Nov 6, 2026

View reviewed changes

Navigation Menu

Repository navigation

MAINT Prefer public package imports and enforce export boundaries - #3848

MAINT Prefer public package imports and enforce export boundaries#3848
Roman Lutz (romanlutz) wants to merge 4 commits into
microsoft:mainfrom
romanlutz:romanlutz-public-package-imports

Roman Lutz (romanlutz) commented Nov 3, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Spencer Schoenberg (spencrr) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Roman Lutz (romanlutz) commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Tests and Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Spencer Schoenberg (spencrr) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Roman Lutz (romanlutz) commented Nov 3, 2026 •

edited

Loading