Skip to content

Commit 414293c

Browse files
gargsaumyagargsaumya
authored
FEAT: Add Azure SQL Database to PR validation pipeline matrix (#428)
### Work Item / Issue Reference <!-- IMPORTANT: Please follow the PR template guidelines below. For mssql-python maintainers: Insert your ADO Work Item ID below For external contributors: Insert Github Issue number below Only one reference is required - either GitHub issue OR ADO Work Item. --> <!-- mssql-python maintainers: ADO Work Item --> > [AB#41879](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/41879) <!-- External contributors: GitHub Issue --> > GitHub Issue: #<ISSUE_NUMBER> ------------------------------------------------------------------- ### Summary <!-- Insert your summary of changes below. Minimum 10 characters required. --> This pull request updates the `eng/pipelines/pr-validation-pipeline.yml` file to streamline Windows testing jobs and introduce new Azure SQL Database testing jobs for Windows, macOS, and Linux environments. The most significant changes are the removal of temporary CodeQL workarounds and the addition of cross-platform jobs for testing against Azure SQL Database. Azure SQL Database testing jobs (new functionality): * Added `PytestOnWindows_AzureSQL`, `PytestOnMacOS_AzureSQL`, and `PytestOnLinux_AzureSQL` jobs to run tests directly against Azure SQL Database, gated by the `ENABLE_AZURE_SQL` variable. These jobs set up dependencies, build bindings, execute tests, and publish results for each platform. Pipeline cleanup (removal of temporary workarounds): * Removed temporary CodeQL variables and matrix entries from the Windows job, which were previously used to clear stale snapshots related to issue SM02986. * Deleted temporary CodeQL initialization and finalization tasks from the Windows job steps, as the workaround is no longer needed. <!-- ### PR Title Guide > For feature requests FEAT: (short-description) > For non-feature requests like test case updates, config updates , dependency updates etc CHORE: (short-description) > For Fix requests FIX: (short-description) > For doc update requests DOC: (short-description) > For Formatting, indentation, or styling update STYLE: (short-description) > For Refactor, without any feature changes REFACTOR: (short-description) > For release related changes, without any feature changes RELEASE: #<RELEASE_VERSION> (short-description) ### Contribution Guidelines External contributors: - Create a GitHub issue first: https://github.com/microsoft/mssql-python/issues/new - Link the GitHub issue in the "GitHub Issue" section above - Follow the PR title format and provide a meaningful summary mssql-python maintainers: - Create an ADO Work Item following internal processes - Link the ADO Work Item in the "ADO Work Item" section above - Follow the PR title format and provide a meaningful summary -->
1 parent 4b28f34 commit 414293c

File tree

1 file changed

+208
-20
lines changed

1 file changed

+208
-20
lines changed

‎eng/pipelines/pr-validation-pipeline.yml‎

Lines changed: 208 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -49,18 +49,9 @@ jobs:
4949
displayName: 'Windows x64'
5050
pool:
5151
vmImage: 'windows-latest'
52-
variables:
53-
# Enable CodeQL for this job to update the old stale snapshot (build_jobname=pytestonwindows)
54-
# This can be removed once the old CodeQL issue SM02986 is cleared
55-
Codeql.Enabled: true
5652

5753
strategy:
5854
matrix:
59-
pytestonwindows:
60-
# Temporary entry to clear stale CodeQL snapshot SM02986
61-
# Remove this once the issue is resolved
62-
sqlVersion: 'SQL2022'
63-
pythonVersion: '3.13'
6455
SQLServer2022:
6556
sqlVersion: 'SQL2022'
6657
pythonVersion: '3.13'
@@ -234,22 +225,11 @@ jobs:
234225
env:
235226
DB_PASSWORD: $(DB_PASSWORD)
236227
237-
# ============== CodeQL Init (temporary - remove after SM02986 is cleared) ==============
238-
- task: CodeQL3000Init@0
239-
inputs:
240-
Enabled: true
241-
displayName: 'Initialize CodeQL (temporary)'
242-
243228
- script: |
244229
cd mssql_python\pybind
245230
build.bat x64
246231
displayName: 'Build .pyd file'
247232
248-
# ============== CodeQL Finalize (temporary - remove after SM02986 is cleared) ==============
249-
- task: CodeQL3000Finalize@0
250-
condition: always()
251-
displayName: 'Finalize CodeQL (temporary)'
252-
253233
# Run tests for LocalDB
254234
- script: |
255235
python -m pytest -v --junitxml=test-results-localdb.xml --cov=. --cov-report=xml:coverage-localdb.xml --capture=tee-sys --cache-clear
@@ -1975,6 +1955,214 @@ jobs:
19751955
testResultsFiles: '**/test-results-alpine-arm64.xml'
19761956
testRunTitle: 'Publish pytest results on Alpine ARM64'
19771957

1958+
# ===========================================================================================
1959+
# Azure SQL Database Testing Jobs
1960+
# Gated on the AZURE_CONNECTION_STRING pipeline variable being set.
1961+
# No local SQL Server install needed — tests run directly against the cloud database.
1962+
# Runs every 16 days (on days divisible by 16) or when ENABLE_AZURE_SQL is set to true
1963+
# ===========================================================================================
1964+
1965+
- job: CheckAzureSQLSchedule
1966+
displayName: 'Check if Azure SQL tests should run'
1967+
pool:
1968+
vmImage: 'ubuntu-latest'
1969+
steps:
1970+
- bash: |
1971+
day=$(date +%d)
1972+
remainder=$((10#$day % 15))
1973+
if [ "$remainder" -eq 0 ] || [ "$(ENABLE_AZURE_SQL)" = "true" ]; then
1974+
echo "##vso[task.setvariable variable=shouldRunAzureSQL;isOutput=true]true"
1975+
echo "Azure SQL tests will run (day: $day, ENABLE_AZURE_SQL: $(ENABLE_AZURE_SQL))"
1976+
else
1977+
echo "##vso[task.setvariable variable=shouldRunAzureSQL;isOutput=true]false"
1978+
echo "Azure SQL tests will be skipped (day: $day, ENABLE_AZURE_SQL: $(ENABLE_AZURE_SQL))"
1979+
fi
1980+
displayName: 'Determine if Azure SQL tests should run'
1981+
name: checkSchedule
1982+
1983+
- job: PytestOnWindows_AzureSQL
1984+
displayName: 'Windows x64 AzureSQL'
1985+
dependsOn: CheckAzureSQLSchedule
1986+
condition: eq(dependencies.CheckAzureSQLSchedule.outputs['checkSchedule.shouldRunAzureSQL'], 'true')
1987+
pool:
1988+
vmImage: 'windows-latest'
1989+
1990+
steps:
1991+
- task: UsePythonVersion@0
1992+
inputs:
1993+
versionSpec: '3.13'
1994+
addToPath: true
1995+
githubToken: $(GITHUB_TOKEN)
1996+
displayName: 'Use Python 3.13'
1997+
1998+
- script: |
1999+
python -m pip install --upgrade pip
2000+
pip install -r requirements.txt
2001+
displayName: 'Install dependencies'
2002+
2003+
- script: |
2004+
cd mssql_python\pybind
2005+
build.bat x64
2006+
displayName: 'Build .pyd file'
2007+
2008+
- script: |
2009+
python -m pytest -v --junitxml=test-results-azuresql.xml --cov=. --cov-report=xml:coverage-azuresql.xml --capture=tee-sys --cache-clear
2010+
displayName: 'Run tests on Azure SQL Database'
2011+
env:
2012+
DB_CONNECTION_STRING: $(AZURE_CONNECTION_STRING)
2013+
2014+
- task: PublishTestResults@2
2015+
condition: succeededOrFailed()
2016+
inputs:
2017+
testResultsFiles: '**/test-results-azuresql.xml'
2018+
testRunTitle: 'Publish test results for Windows AzureSQL'
2019+
2020+
- job: PytestOnMacOS_AzureSQL
2021+
displayName: 'macOS x86_64 AzureSQL'
2022+
dependsOn: CheckAzureSQLSchedule
2023+
condition: eq(dependencies.CheckAzureSQLSchedule.outputs['checkSchedule.shouldRunAzureSQL'], 'true')
2024+
pool:
2025+
vmImage: 'macos-latest'
2026+
2027+
steps:
2028+
- task: UsePythonVersion@0
2029+
inputs:
2030+
versionSpec: '3.13'
2031+
addToPath: true
2032+
displayName: 'Use Python 3.13'
2033+
2034+
- script: |
2035+
brew update
2036+
brew uninstall cmake --ignore-dependencies || echo "CMake not installed or already removed"
2037+
brew install cmake
2038+
displayName: 'Install CMake'
2039+
2040+
- script: |
2041+
python -m pip install --upgrade pip
2042+
pip install -r requirements.txt
2043+
displayName: 'Install Python dependencies'
2044+
2045+
- script: |
2046+
cd mssql_python/pybind
2047+
./build.sh
2048+
displayName: 'Build pybind bindings (.so)'
2049+
2050+
- script: |
2051+
python -m pytest -v --junitxml=test-results-azuresql.xml --cov=. --cov-report=xml:coverage-azuresql.xml --capture=tee-sys --cache-clear
2052+
displayName: 'Run tests on Azure SQL Database'
2053+
env:
2054+
DB_CONNECTION_STRING: $(AZURE_CONNECTION_STRING)
2055+
2056+
- task: PublishTestResults@2
2057+
condition: succeededOrFailed()
2058+
inputs:
2059+
testResultsFiles: '**/test-results-azuresql.xml'
2060+
testRunTitle: 'Publish pytest results on macOS AzureSQL'
2061+
2062+
- job: PytestOnLinux_AzureSQL
2063+
displayName: 'Linux x86_64 AzureSQL'
2064+
dependsOn: CheckAzureSQLSchedule
2065+
condition: eq(dependencies.CheckAzureSQLSchedule.outputs['checkSchedule.shouldRunAzureSQL'], 'true')
2066+
pool:
2067+
vmImage: 'ubuntu-latest'
2068+
2069+
steps:
2070+
- script: |
2071+
docker run -d --name test-container-ubuntu-azuresql \
2072+
-v $(Build.SourcesDirectory):/workspace \
2073+
-w /workspace \
2074+
--network bridge \
2075+
ubuntu:22.04 \
2076+
tail -f /dev/null
2077+
displayName: 'Create Ubuntu container'
2078+
2079+
- script: |
2080+
docker exec test-container-ubuntu-azuresql bash -c "
2081+
export DEBIAN_FRONTEND=noninteractive
2082+
export TZ=UTC
2083+
ln -snf /usr/share/zoneinfo/\$TZ /etc/localtime && echo \$TZ > /etc/timezone
2084+
apt-get update &&
2085+
apt-get install -y python3 python3-pip python3-venv python3-full cmake curl wget gnupg software-properties-common build-essential python3-dev pybind11-dev
2086+
"
2087+
displayName: 'Install basic dependencies in Ubuntu container'
2088+
2089+
- script: |
2090+
docker exec test-container-ubuntu-azuresql bash -c "
2091+
export DEBIAN_FRONTEND=noninteractive
2092+
curl -sSL -O https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb
2093+
dpkg -i packages-microsoft-prod.deb || true
2094+
rm packages-microsoft-prod.deb
2095+
apt-get update
2096+
ACCEPT_EULA=Y apt-get install -y msodbcsql18
2097+
ACCEPT_EULA=Y apt-get install -y mssql-tools18
2098+
apt-get install -y unixodbc-dev
2099+
"
2100+
displayName: 'Install ODBC Driver in Ubuntu container'
2101+
2102+
- script: |
2103+
docker exec test-container-ubuntu-azuresql bash -c "
2104+
python3 -m venv /opt/venv
2105+
source /opt/venv/bin/activate
2106+
python -m pip install --upgrade pip
2107+
python -m pip install -r requirements.txt
2108+
echo 'source /opt/venv/bin/activate' >> ~/.bashrc
2109+
"
2110+
displayName: 'Install Python dependencies in Ubuntu container'
2111+
2112+
- script: |
2113+
docker exec test-container-ubuntu-azuresql bash -c "
2114+
source /opt/venv/bin/activate
2115+
cd mssql_python/pybind
2116+
chmod +x build.sh
2117+
./build.sh
2118+
"
2119+
displayName: 'Build pybind bindings (.so) in Ubuntu container'
2120+
2121+
- script: |
2122+
docker exec test-container-ubuntu-azuresql bash -c "
2123+
export DEBIAN_FRONTEND=noninteractive
2124+
apt-get remove --purge -y msodbcsql18 mssql-tools18 unixodbc-dev
2125+
rm -f /usr/bin/sqlcmd
2126+
rm -f /usr/bin/bcp
2127+
rm -rf /opt/microsoft/msodbcsql
2128+
rm -f /lib/x86_64-linux-gnu/libodbcinst.so.2
2129+
odbcinst -u -d -n 'ODBC Driver 18 for SQL Server' || true
2130+
echo 'Uninstalled ODBC Driver and cleaned up libraries'
2131+
echo 'Verifying x86_64 debian_ubuntu driver library signatures:'
2132+
ldd mssql_python/libs/linux/debian_ubuntu/x86_64/lib/libmsodbcsql-18.5.so.1.1
2133+
"
2134+
displayName: 'Uninstall ODBC Driver before running tests in Ubuntu container'
2135+
2136+
- script: |
2137+
docker exec \
2138+
-e DB_CONNECTION_STRING="$(AZURE_CONNECTION_STRING)" \
2139+
test-container-ubuntu-azuresql bash -c "
2140+
source /opt/venv/bin/activate
2141+
echo 'Running tests on Ubuntu against Azure SQL Database'
2142+
python -m pytest -v --junitxml=test-results-ubuntu-azuresql.xml --cov=. --cov-report=xml:coverage-ubuntu-azuresql.xml --capture=tee-sys --cache-clear
2143+
"
2144+
displayName: 'Run pytest on Azure SQL in Ubuntu container'
2145+
env:
2146+
AZURE_CONNECTION_STRING: $(AZURE_CONNECTION_STRING)
2147+
2148+
- script: |
2149+
docker cp test-container-ubuntu-azuresql:/workspace/test-results-ubuntu-azuresql.xml $(Build.SourcesDirectory)/
2150+
docker cp test-container-ubuntu-azuresql:/workspace/coverage-ubuntu-azuresql.xml $(Build.SourcesDirectory)/
2151+
displayName: 'Copy test results from Ubuntu container'
2152+
condition: always()
2153+
2154+
- script: |
2155+
docker stop test-container-ubuntu-azuresql || true
2156+
docker rm test-container-ubuntu-azuresql || true
2157+
displayName: 'Clean up Ubuntu container'
2158+
condition: always()
2159+
2160+
- task: PublishTestResults@2
2161+
condition: succeededOrFailed()
2162+
inputs:
2163+
testResultsFiles: '**/test-results-ubuntu-azuresql.xml'
2164+
testRunTitle: 'Publish pytest results on Ubuntu AzureSQL'
2165+
19782166
- job: CodeCoverageReport
19792167
displayName: 'Full Code Coverage Report in Ubuntu x86_64'
19802168
pool:

0 commit comments

Comments
 (0)