Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
02911ed
[#154] Added Google Drive database fetch and push scripts and the 'vo…
AlexSkrypnyk Nov 10, 2026
e1aeae9
[#154] Fixed mock expectations in the Google Drive and refresh tests.
AlexSkrypnyk Nov 10, 2026
6848883
[#154] Documented pushing the sanitized database to a Google Drive or…
AlexSkrypnyk Nov 10, 2026
86b9445
Updated snapshots.
AlexSkrypnyk Nov 10, 2026
1318811
[#154] Fetched the dump from the upstream source in 'vortex-fetch-db'…
AlexSkrypnyk Nov 10, 2026
54192c1
[#154] Narrowed the fetch-db upstream source failure assertion.
AlexSkrypnyk Nov 10, 2026
dfc4cbf
[#154] Regenerated the variables reference.
AlexSkrypnyk Nov 10, 2026
40bc12e
Updated snapshots.
AlexSkrypnyk Nov 10, 2026
7f433eb
Addressed code review: defaulted the Google Drive and S3 push file na…
AlexSkrypnyk Nov 11, 2026
d0d45e0
Addressed code review: regenerated the variables reference.
AlexSkrypnyk Nov 11, 2026
5da3fa9
Replaced the Google Drive push with an FTP push that keeps the curren…
AlexSkrypnyk Nov 11, 2026
abc3c9b
Renamed the S3 push test loop variable that clashed with the BATS sta…
AlexSkrypnyk Nov 11, 2026
ff1e3bc
Wired the FTP and S3 dump store push into CI and the installer, and d…
AlexSkrypnyk Nov 11, 2026
9bdd6f7
Updated snapshots.
AlexSkrypnyk Nov 11, 2026
a52794b
Pushed the latest dump under the remote file name with an optional su…
AlexSkrypnyk Nov 11, 2026
ba14e7f
Updated the database storage docs and the generated CI configs for th…
AlexSkrypnyk Nov 11, 2026
185a3c5
Updated snapshots.
AlexSkrypnyk Nov 11, 2026
095278e
Addressed code review: sent an empty length on the S3 copy request an…
AlexSkrypnyk Nov 11, 2026
9a9a2ff
Addressed code review: documented setting separate push and fetch fil…
AlexSkrypnyk Nov 11, 2026
728cb8c
Addressed code review: documented renaming the dump with 'VORTEX_DB_F…
AlexSkrypnyk Nov 11, 2026
1289fc0
Reworded the suffixed S3 fetch note to pass the docs spellcheck.
AlexSkrypnyk Nov 11, 2026
2ad5b39
Added a playground script to fetch the database dump from FTP.
AlexSkrypnyk Nov 11, 2026
71eb4e0
Added the 'VORTEX_FETCH_DB_FTP_DIR' remote directory for the FTP fetc…
AlexSkrypnyk Nov 11, 2026
032e31f
Added 'VORTEX_PUSH_DB_DESTINATIONS' to push the database dump to seve…
AlexSkrypnyk Nov 11, 2026
6f85d7c
Documented the FTP directory and pushing to several destinations.
AlexSkrypnyk Nov 11, 2026
0ea15e8
Updated snapshots.
AlexSkrypnyk Nov 11, 2026
f081d65
Reworded the timestamped copy note in '.env' so the configured timezo…
AlexSkrypnyk Nov 11, 2026
f858aec
Updated snapshots.
AlexSkrypnyk Nov 11, 2026
ed3495c
Pushed Vortex's own demo databases to the destinations in the 'VORTEX…
AlexSkrypnyk Nov 15, 2026
32ac419
Used 'filename' in the database storage docs to pass the prose check.
AlexSkrypnyk Nov 15, 2026
4710c0e
Updated snapshots.
AlexSkrypnyk Nov 15, 2026
321d3fd
Re-triggered CI to clear a flaky check.
AlexSkrypnyk Nov 15, 2026
5a5b228
[#3340] Added 'VORTEX_FETCH_DB_STORE' to fetch the database from a sa…
AlexSkrypnyk Nov 16, 2026
327052b
[#3340] Documented fetching from a sanitized store, refreshing it on …
AlexSkrypnyk Nov 16, 2026
d820dfd
[#3340] Added the 'Database store' and 'Migration database store' ins…
AlexSkrypnyk Nov 16, 2026
a78110a
Updated snapshots.
AlexSkrypnyk Nov 16, 2026
805e478
Updated snapshots.
AlexSkrypnyk Nov 16, 2026
71a408f
[#3340] Split the deploy job condition over 2 lines to stay within th…
AlexSkrypnyk Nov 16, 2026
38c90a4
Updated snapshots.
AlexSkrypnyk Nov 16, 2026
de61aef
[#3340] Re-recorded the installer video with the database store prompt.
AlexSkrypnyk Nov 16, 2026
a657bdd
[#3340] Read the 'refresh_db' input from the event payload in the ste…
AlexSkrypnyk Nov 16, 2026
1b9f0bc
Updated snapshots.
AlexSkrypnyk Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,15 @@
#; and will be removed during installation or update of Vortex.
version: '2.1'

#;< !PROVISION_TYPE_PROFILE
parameters:
# Refresh the database as on the nightly run, without building or deploying.
# Set it when triggering a pipeline from the CircleCI UI or API.
refresh_db:
type: boolean
default: false
#;> !PROVISION_TYPE_PROFILE

aliases:
#;< !PROVISION_TYPE_PROFILE
# SSH key fingerprint to fetch the database.
Expand Down Expand Up @@ -363,6 +372,27 @@ jobs:
no_output_timeout: 30m
#;> DB_FETCH_SOURCE_CONTAINER_REGISTRY

# Push the freshly-exported sanitized database dump to the push
# destinations. No-op unless a fresh DB was fetched (the fetch semaphore) and
# the push was requested (VORTEX_PUSH_DB_PROCEED, in the script).
- run:
name: Push DB
command: |
[ ! -f /tmp/fetch-db-fresh ] && echo "==> No freshly fetched database. DB push will not proceed." && exit 0
./vendor/bin/vortex-push-db
no_output_timeout: 30m

#;< MIGRATION
# Push the freshly-fetched migration database dump to its push destinations,
# next to the primary dump. No-op unless the push was requested.
- run:
name: Push migration DB
command: |
[ ! -f /tmp/fetch-db-fresh ] && echo "==> No freshly fetched database. DB push will not proceed." && exit 0
VORTEX_DB_INDEX=2 ./vendor/bin/vortex-push-db
no_output_timeout: 30m
#;> MIGRATION

- save_cache:
# Save cache per default branch and the timestamp.
# The cache will not be saved if it already exists.
Expand All @@ -387,6 +417,17 @@ jobs:
# Deploy container image.
VORTEX_EXPORT_DB_CONTAINER_REGISTRY_PUSH_PROCEED: 1
#;> DB_IN_IMAGE
# Push the sanitized database dump to the database store and destinations.
VORTEX_PUSH_DB_PROCEED: 1
#;< VORTEX_DEV
# Tell this provider's demo database dumps apart on the test FTP server
# by directory, with a timestamped copy of each push. The destinations
# and the FTP server come from the VORTEX_PUSH_DB_DESTINATIONS,
# VORTEX_PUSH_DB2_DESTINATIONS and VORTEX_PUSH_DB_FTP_* project
# variables.
VORTEX_PUSH_DB_FTP_DIR: circleci
VORTEX_PUSH_DB_TIMESTAMPED_COPY: 1
#;> VORTEX_DEV
# Do not build the Drupal front-end.
VORTEX_FRONTEND_BUILD_SKIP: 1
#;> !PROVISION_TYPE_PROFILE
Expand Down Expand Up @@ -759,6 +800,10 @@ jobs:
workflows:
# Commit workflow. Runs for every commit push to the remote repository.
commit:
#;< !PROVISION_TYPE_PROFILE
when:
not: << pipeline.parameters.refresh_db >>
#;> !PROVISION_TYPE_PROFILE
jobs:
#;< !PROVISION_TYPE_PROFILE
- database:
Expand Down Expand Up @@ -859,4 +904,11 @@ workflows:
- develop
jobs:
- database-nightly

# Database refresh workflow runs the nightly database job on demand, when a
# pipeline is triggered with the "refresh_db" parameter.
refresh-db:
when: << pipeline.parameters.refresh_db >>
jobs:
- database-nightly
#;> !PROVISION_TYPE_PROFILE
21 changes: 21 additions & 0 deletions .circleci/vortex-test-common.yml
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,27 @@ jobs:
no_output_timeout: 30m
#;> DB_FETCH_SOURCE_CONTAINER_REGISTRY

# Push the freshly-exported sanitized database dump to the push
# destinations. No-op unless a fresh DB was fetched (the fetch semaphore) and
# the push was requested (VORTEX_PUSH_DB_PROCEED, in the script).
- run:
name: Push DB
command: |
[ ! -f /tmp/fetch-db-fresh ] && echo "==> No freshly fetched database. DB push will not proceed." && exit 0
./vendor/bin/vortex-push-db
no_output_timeout: 30m

#;< MIGRATION
# Push the freshly-fetched migration database dump to its push destinations,
# next to the primary dump. No-op unless the push was requested.
- run:
name: Push migration DB
command: |
[ ! -f /tmp/fetch-db-fresh ] && echo "==> No freshly fetched database. DB push will not proceed." && exit 0
VORTEX_DB_INDEX=2 ./vendor/bin/vortex-push-db
no_output_timeout: 30m
#;> MIGRATION

- save_cache:
# Save cache per default branch and the timestamp.
# The cache will not be saved if it already exists.
Expand Down
40 changes: 40 additions & 0 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,11 @@ VORTEX_FETCH_DB_FTP_HOST=
# Database dump FTP port.
VORTEX_FETCH_DB_FTP_PORT=21

# Database dump FTP directory.
#
# Leave empty for the root directory of the FTP account.
VORTEX_FETCH_DB_FTP_DIR=

# Database dump FTP file name.
VORTEX_FETCH_DB_FTP_FILE=db.sql
#;> DB_FETCH_SOURCE_FTP
Expand All @@ -258,6 +263,24 @@ VORTEX_FETCH_DB_S3_BUCKET=
VORTEX_FETCH_DB_S3_REGION=ap-southeast-2
#;> DB_FETCH_SOURCE_S3

# Database store of the sanitized database dump file: ftp or s3.
#
# When set, the database is fetched from the store instead of the source. The
# nightly CI job fetches from the source, sanitizes the database and pushes
# the dump to the store. The store reads the FTP or S3 variables above. Leave
# empty to fetch from the source.
VORTEX_FETCH_DB_STORE=

# Destinations the nightly CI job pushes the sanitized database dump to
# besides the database store: ftp, s3 or both, such as "ftp,s3". Each
# destination reads its own variables, such as VORTEX_PUSH_DB_FTP_HOST for ftp.
# The database is always fetched from 1 place.
# VORTEX_PUSH_DB_DESTINATIONS=ftp,s3

# Also push a copy of each dump named with the time of the push, such as
# db_20261111-180000.sql, to keep a history of the pushed dumps.
# VORTEX_PUSH_DB_TIMESTAMPED_COPY=1

# Environment to fetch the database from.
#
# Applies to hosting environments.
Expand Down Expand Up @@ -319,6 +342,11 @@ VORTEX_FETCH_DB2_FTP_HOST=
# Second database dump FTP port.
VORTEX_FETCH_DB2_FTP_PORT=21

# Second database dump FTP directory.
#
# Leave empty for the root directory of the FTP account.
VORTEX_FETCH_DB2_FTP_DIR=

# Second database dump FTP file name.
VORTEX_FETCH_DB2_FTP_FILE=db2.sql
#;> MIGRATION_DB_FETCH_SOURCE_FTP
Expand All @@ -333,6 +361,18 @@ VORTEX_FETCH_DB2_S3_BUCKET=
VORTEX_FETCH_DB2_S3_REGION=ap-southeast-2
#;> MIGRATION_DB_FETCH_SOURCE_S3

# Database store of the second database dump file: ftp or s3.
#
# When set, the second database is fetched from the store instead of the
# source. The nightly CI job fetches from the source and pushes the dump, as
# fetched, to the store, next to the primary dump. Leave empty to fetch from
# the source.
VORTEX_FETCH_DB2_STORE=

# Destinations the nightly CI job pushes the second database dump to, as
# fetched, besides the database store: ftp, s3 or both.
# VORTEX_PUSH_DB2_DESTINATIONS=ftp,s3

# Environment to fetch the second database from.
#
# Applies to hosting environments.
Expand Down
70 changes: 66 additions & 4 deletions .github/workflows/build-test-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,13 @@ on:
description: 'Enable terminal session for CI jobs'
required: false
default: false
#;< !PROVISION_TYPE_PROFILE
refresh_db:
type: boolean
description: 'Refresh the database only'
required: false
default: false
#;> !PROVISION_TYPE_PROFILE
#;< !PROVISION_TYPE_PROFILE
schedule:
- cron: '0 18 * * *'
Expand All @@ -84,7 +91,7 @@ jobs:
lint:
runs-on: ubuntu-latest
#;< !PROVISION_TYPE_PROFILE
if: ${{ !inputs.deploy_target && github.event_name != 'schedule' && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')) }}
if: ${{ !inputs.deploy_target && !inputs.refresh_db && github.event_name != 'schedule' && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')) }}
#;> !PROVISION_TYPE_PROFILE

container:
Expand Down Expand Up @@ -225,6 +232,38 @@ jobs:
VORTEX_CI_DB_CACHE_FALLBACK: "yes"
# Which branch to use as a source of DB caches.
VORTEX_CI_DB_CACHE_BRANCH: "develop"
#;< DB_FETCH_SOURCE_FTP
VORTEX_FETCH_DB_FTP_USER: ${{ secrets.VORTEX_FETCH_DB_FTP_USER }}
VORTEX_FETCH_DB_FTP_PASS: ${{ secrets.VORTEX_FETCH_DB_FTP_PASS }}
#;> DB_FETCH_SOURCE_FTP
#;< DB_FETCH_SOURCE_S3
VORTEX_FETCH_DB_S3_ACCESS_KEY: ${{ secrets.VORTEX_FETCH_DB_S3_ACCESS_KEY }}
VORTEX_FETCH_DB_S3_SECRET_KEY: ${{ secrets.VORTEX_FETCH_DB_S3_SECRET_KEY }}
#;> DB_FETCH_SOURCE_S3
#;< MIGRATION_DB_FETCH_SOURCE_FTP
VORTEX_FETCH_DB2_FTP_USER: ${{ secrets.VORTEX_FETCH_DB2_FTP_USER }}
VORTEX_FETCH_DB2_FTP_PASS: ${{ secrets.VORTEX_FETCH_DB2_FTP_PASS }}
#;> MIGRATION_DB_FETCH_SOURCE_FTP
#;< MIGRATION_DB_FETCH_SOURCE_S3
VORTEX_FETCH_DB2_S3_ACCESS_KEY: ${{ secrets.VORTEX_FETCH_DB2_S3_ACCESS_KEY }}
VORTEX_FETCH_DB2_S3_SECRET_KEY: ${{ secrets.VORTEX_FETCH_DB2_S3_SECRET_KEY }}
#;> MIGRATION_DB_FETCH_SOURCE_S3
#;< VORTEX_DEV
# Push destinations of the demo databases on the nightly run. Empty
# repository variables skip the push.
VORTEX_PUSH_DB_DESTINATIONS: ${{ vars.VORTEX_PUSH_DB_DESTINATIONS }}
VORTEX_PUSH_DB2_DESTINATIONS: ${{ vars.VORTEX_PUSH_DB2_DESTINATIONS }}
# FTP server that receives both demo databases, in a directory that
# tells this provider's dumps apart, with a timestamped copy of each
# push.
VORTEX_PUSH_DB_FTP_HOST: ${{ secrets.VORTEX_PUSH_DB_FTP_HOST }}
VORTEX_PUSH_DB_FTP_PORT: ${{ secrets.VORTEX_PUSH_DB_FTP_PORT }}
VORTEX_PUSH_DB_FTP_USER: ${{ secrets.VORTEX_PUSH_DB_FTP_USER }}
VORTEX_PUSH_DB_FTP_PASS: ${{ secrets.VORTEX_PUSH_DB_FTP_PASS }}
VORTEX_PUSH_DB_FTP_DIR: ${{ vars.VORTEX_PUSH_DB_FTP_DIR || 'gha' }}
VORTEX_PUSH_DB_FTP_FILE: ${{ vars.VORTEX_PUSH_DB_FTP_FILE || 'db.sql' }}
VORTEX_PUSH_DB_TIMESTAMPED_COPY: 1
#;> VORTEX_DEV

steps:
# Frees disk space by removing preinstalled toolchains unused by this project. Steps run in a container, so the host is reached via the Docker socket.
Expand Down Expand Up @@ -270,7 +309,7 @@ jobs:
run: ./scripts/vortex-tooling.sh

- name: Adjust variables for a scheduled run
if: github.event_name == 'schedule'
if: github.event_name == 'schedule' || github.event.inputs.refresh_db == 'true'
run: |
{
echo "VORTEX_CI_DB_CACHE_FALLBACK=no"
Expand All @@ -281,6 +320,8 @@ jobs:
# Deploy container image.
echo "VORTEX_EXPORT_DB_CONTAINER_REGISTRY_PUSH_PROCEED=1"
#;> DB_IN_IMAGE
# Push the sanitized database dump to the database store and destinations.
echo "VORTEX_PUSH_DB_PROCEED=1"
} >> "$GITHUB_ENV"

- name: Create cache keys files for database caching
Expand Down Expand Up @@ -343,6 +384,25 @@ jobs:
timeout-minutes: 30
#;> DB_FETCH_SOURCE_CONTAINER_REGISTRY

# Push the freshly-exported sanitized database dump to the push
# destinations. No-op unless a fresh DB was fetched (the fetch semaphore) and
# the push was requested (VORTEX_PUSH_DB_PROCEED, in the script).
- name: Push DB
run: |
if [ ! -f /tmp/fetch-db-fresh ]; then echo "==> No freshly fetched database. DB push will not proceed."; exit 0; fi
./vendor/bin/vortex-push-db
timeout-minutes: 30

#;< MIGRATION
# Push the freshly-fetched migration database dump to its push destinations,
# next to the primary dump. No-op unless the push was requested.
- name: Push migration DB
run: |
if [ ! -f /tmp/fetch-db-fresh ]; then echo "==> No freshly fetched database. DB push will not proceed."; exit 0; fi
VORTEX_DB_INDEX=2 ./vendor/bin/vortex-push-db
timeout-minutes: 30
#;> MIGRATION

# Save cache per default branch and the timestamp.
# The cache will not be saved if it already exists.
# Note that the cache fallback flag is enabled for this case in order
Expand All @@ -359,7 +419,7 @@ jobs:
runs-on: ubuntu-latest
#;< !PROVISION_TYPE_PROFILE
needs: database
if: ${{ !inputs.deploy_target && github.event_name != 'schedule' && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')) }}
if: ${{ !inputs.deploy_target && !inputs.refresh_db && github.event_name != 'schedule' && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')) }}
#;> !PROVISION_TYPE_PROFILE

permissions:
Expand Down Expand Up @@ -717,7 +777,9 @@ jobs:
runs-on: ubuntu-latest
needs: [build, lint]
#;< !PROVISION_TYPE_PROFILE
if: ${{ !cancelled() && (inputs.deploy_target || (success() && github.event_name != 'schedule' && !startsWith(github.head_ref || github.ref_name, 'deps/') && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')))) }}
if: >-
${{ !cancelled() && (inputs.deploy_target || (success() && !inputs.refresh_db && github.event_name != 'schedule'
&& !startsWith(github.head_ref || github.ref_name, 'deps/') && (github.event_name == 'push' || !startsWith(github.head_ref, 'project/')))) }}
#;> !PROVISION_TYPE_PROFILE

permissions:
Expand Down
Loading
Loading