Skip to content

Commit a55754a

Browse files
Merging all V1.1 changes into master (#3)
* fixing dead link * shortening links * minor text changes * dst timezone fix * Adding serverless job types * Adding display name * fixing serverless pool type * ordering job pool properties * fix for empty step sections * adding pipeline and job demands for non serverless agent jobs * added blog posts * updated docs to reflect changes in demands * changing the way how jobs are ID-ed and reffed by in demands * fixing duplication of demands in the case of multiple jobs * commenting emptying construct for steps * documentation on changing job id and build number * re-ordering triggers * fixing wrong syntax for included / excluded when multiple found * strong typing some arrays to include arrayed single / null tasks * more strong-typing * fixing job properties for time-outs * camelcasing for timeouts * adding multi sliced ToDO * strongtyping returned array and remove from child call * adding checkout properties * adding 1.1 doc changes except resources * adding pipeline wide timeouts * finalizing v1.1 docs * updating PSD
1 parent 4664eb7 commit a55754a

11 files changed

Lines changed: 222 additions & 80 deletions

README.md

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,38 @@
11
# AzDoAPITools
22

3-
[![Build Status](https://dev.azure.com/ContinuousData/cdtestproject/_apis/build/status/tsteenbakkers.AzDoAPITools?branchName=master)](https://dev.azure.com/ContinuousData/cdtestproject/_build/latest?definitionId=4&branchName=master)
4-
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/Continuous-Data/AzDoAPITools/blob/master/LICENSE)
5-
[![Documentation - AzDoAPITools](https://img.shields.io/badge/Documentation-AzDoAPITools-blue.svg)](https://github.com/Continuous-Data/AzDoAPITools/blob/master/docs/readme.md)
3+
[![Build Status](https://dev.azure.com/ContinuousData/cdtestproject/_apis/build/status/tsteenbakkers.AzDoAPITools?repoName=Continuous-Data%2FAzDoAPITools&branchName=master)](https://dev.azure.com/ContinuousData/cdtestproject/_build/latest?definitionId=4&repoName=Continuous-Data%2FAzDoAPITools&branchName=master)
4+
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE.md)
5+
[![Documentation - AzDoAPITools](https://img.shields.io/badge/Documentation-AzDoAPITools-blue.svg)](./docs/README.md)
66
[![PowerShell Gallery - AzDoAPITools](https://img.shields.io/badge/PowerShell%20Gallery-AzDoAPITools-blue.svg)](https://www.powershellgallery.com/packages/AzDoAPITools)
77
[![Minimum Supported PowerShell Version](https://img.shields.io/badge/PowerShell-5.1-blue.svg)](https://github.com/PowerShell/PowerShell)
88

9+
## Version 1.1 Changes
10+
11+
This first update to AzDoAPITools was inspired by Microsofts own [Export to YAML](https://devblogs.microsoft.com/devops/replacing-view-yaml/) functionality introduced in [sprint 178](https://docs.microsoft.com/en-us/azure/devops/release-notes/2020/sprint-178-update#improve-yaml-conversion-in-the-classic-build-designer) inspired me to update my own tooling. After seeing some of the YAML generated by Microsoft I noticed that my tool needed some polishing to make it more in line to the native functionality. I added some ToDo's from the already existing list and added some functionality which I had not taken into account previously. With this update AzDoAPITools is up on par if not better than Microsofts functionality. I have done a comparisson with version 1.0 of AzDoAPITools vs Microsoft "Export to YAML" in [my blog](https://www.continuous-data.nl/azure-devops/full-review-of-the-new-export-to-yaml-feature-in-azure-devops/).
12+
13+
### changes
14+
15+
- added Pipeline / Job demands
16+
- Added pipeline timeouts to every job
17+
- Added Step Properties (checkout, persistcredentials, lfs, submodules etc...)
18+
- added Agentless pools
19+
- Changed Job id / displayName notation (added displayname and use actual API id as id instead of displayName as id)
20+
- Changed order of triggers according to MS Export Tool ordering
21+
- Changes order of job properties to MS Export Tool ordering
22+
- fixed syntax for included / excluded paths and branches notation
23+
- Fixed rare occurrence where executor of script would happen to be in a DST timezone where the target zone would not be.
24+
- Fixed empty jobs syntax. will now be converted as empty array [] rather than a started array.
25+
- Fixed notation for job properties for timeout and canceltimeout.
26+
- updated documentation
27+
928
## Introduction
1029

11-
AzDoAPITools is a project which was born when doing a migration from classical pipelines to YAML pipelines for a customer. Which is the current function of the published module. The module will convert Task Groups and classical build pipelines to usable all in one YAML pipelines / step templates.
30+
AzDoAPITools is a project which was born while doing a migration from classical pipelines (GUI-based) to YAML pipelines (Pipeline as code) for a customer. Which is the current function of the published module. The module will convert Task Groups and classical build pipelines to usable all in one YAML pipelines / step templates.
1231

1332
In the future you can expect other automations which i have done for customers such as automatic branching / mass policy application etc. to be bundled in this module.
1433

34+
In [this blog post](https://www.continuous-data.nl/tools/intro-azdoapitools/) I have written an introduction into this tool and in [this blog post](https://www.continuous-data.nl/azure-devops/full-review-of-the-new-export-to-yaml-feature-in-azure-devops/) there is a comparison with the in-built "Export to YAML" functionality which was created by Microsoft in November 2020.
35+
1536
## Requirements
1637

1738
- Powershell 5.1

Source/AzdoAPITools.psd1

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
RootModule = 'AzdoAPITools.psm1'
1313

1414
# Version number of this module.
15-
ModuleVersion = '1.0.0'
15+
ModuleVersion = '1.1.0'
1616

1717
# Supported PSEditions
1818
# CompatiblePSEditions = @()

Source/Private/Convert-TaskStepsToYAMLSteps.ps1

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ function Convert-TaskStepsToYAMLSteps {
3737

3838
if($steps.count -ge 1){
3939
$taskscount = $input.count
40-
$convertedsteps = @()
40+
[array]$convertedsteps = @()
4141
$convertedcount = 0
4242
Write-verbose "Found $taskscount tasks"
4343
foreach ($step in $steps) {
@@ -73,15 +73,15 @@ function Convert-TaskStepsToYAMLSteps {
7373
}
7474

7575
#### Adding Step to Steps
76-
$convertedsteps += $yamlstep
76+
[array]$convertedsteps += $yamlstep
7777

7878
}elseif($step.task.definitionType -eq 'metaTask' -and $step.enabled -eq 'true'){
7979

8080
$TGtemplate = Get-AzDoAPIToolsDefinitionsTaskGroupsByID -ID $stepid -TGVersion $stepversion -ApiType 'TaskGroup' -Projectname $projectname -profilename $profilename
8181

8282
if ($ExpandNestedTaskGroups.IsPresent) {
83-
$nestedtaskgrouptasks = Convert-TaskStepsToYAMLSteps -profilename $profilename -Projectname $projectname -InputArray $TGTemplate -ExpandNestedTaskGroups -inputType $tgtemplate.type -parentinputtype $inputtype
84-
$convertedsteps += $nestedtaskgrouptasks
83+
[array]$nestedtaskgrouptasks = Convert-TaskStepsToYAMLSteps -profilename $profilename -Projectname $projectname -InputArray $TGTemplate -ExpandNestedTaskGroups -inputType $tgtemplate.type -parentinputtype $inputtype
84+
[array]$convertedsteps += $nestedtaskgrouptasks
8585
}else {
8686

8787
$TGTemplateName = "$($TGTemplate.name).yml"
@@ -95,7 +95,7 @@ function Convert-TaskStepsToYAMLSteps {
9595

9696
}
9797

98-
$convertedsteps += $yamlstep
98+
[array]$convertedsteps += $yamlstep
9999
}
100100

101101
}
@@ -106,8 +106,10 @@ function Convert-TaskStepsToYAMLSteps {
106106
}else{
107107
Write-Verbose 'No Tasks found. skipping steps.'
108108
}
109-
110-
return $convertedsteps
109+
if ($convertedsteps.Count -lt 1) {
110+
[array]$convertedsteps = @()
111+
}
112+
return [array]$convertedsteps
111113
}
112114

113115
}

Source/Private/Get-AzDoAPIToolsAgentPool.ps1

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ function Get-AzDoAPIToolsAgentPool {
1313

1414

1515
process {
16-
$returnedpool = @{}
16+
$returnedpool = [ordered]@{}
1717
$pool = Use-AzDoAPI -url $PoolURL -method 'get'
1818

1919
if ($pool.pool.isHosted -eq 'true') {

Source/Private/Get-CronFromSchedule.ps1

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ function Get-CronFromSchedule {
4444

4545
$SourceTimezone = [System.TimeZoneInfo]::FindSystemTimeZoneById($schedule.timezoneid)
4646

47-
if($SourceTimezone.SupportsDaylightSavingTime -eq $true){
47+
if($SourceTimezone.SupportsDaylightSavingTime -eq $true -and $datetimetoconvert.IsDaylightSavingTime() -eq $true){
4848
$datetimetoconvert = $datetimetoconvert.AddHours(1)
4949
}
5050

Source/Private/Get-DefinitionInputIncludeExclude.ps1

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ function Get-DefinitionInputIncludeExclude {
99
begin{
1010
$included = @()
1111
$excluded = @()
12-
$return = @()
12+
$return = [ordered]@{}
1313
$regex = '^([\+\-])[\\\/]?(.+)'
1414
}
1515

@@ -30,17 +30,11 @@ function Get-DefinitionInputIncludeExclude {
3030
end{
3131

3232
if ($included.count -ge 1) {
33-
$includedinput = @{
34-
'include' = $included
35-
}
36-
$return += $includedinput
33+
$return.add('include',$included)
3734
}
3835

3936
if ($excluded.count -ge 1) {
40-
$excludedinput = @{
41-
'exclude' = $excluded
42-
}
43-
$return += $excludedinput
37+
$return.add('exclude',$excluded)
4438
}
4539

4640
return $return

Source/Private/Get-TaskProperties.ps1

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,43 @@ function Get-TaskProperties {
3838
$FilteredTaskProperties | Add-Member -NotePropertyName $propertyname -NotePropertyValue $propertyvalue
3939
}
4040
}
41+
jobTimeoutInMinutes {
42+
if($propertyvalue -ne 0){
43+
$FilteredTaskProperties | Add-Member -NotePropertyName 'timeoutInMinutes' -NotePropertyValue $propertyvalue
44+
}
45+
}
46+
jobCancelTimeoutInMinutes {
47+
if($propertyvalue -ne 0){
48+
$FilteredTaskProperties | Add-Member -NotePropertyName 'cancelTimeoutInMinutes' -NotePropertyValue $propertyvalue
49+
}
50+
}
51+
clean {
52+
if($propertyvalue -ne $false){
53+
$FilteredTaskProperties | Add-Member -NotePropertyName $propertyname -NotePropertyValue $propertyvalue
54+
}
55+
}
56+
gitLfsSupport {
57+
if($propertyvalue -ne $false){
58+
$FilteredTaskProperties | Add-Member -NotePropertyName 'lfs' -NotePropertyValue $propertyvalue
59+
}
60+
}
61+
checkoutSubmodules {
62+
if($propertyvalue -ne $false){
63+
$FilteredTaskProperties | Add-Member -NotePropertyName 'submodules' -NotePropertyValue $propertyvalue
64+
}
65+
}
66+
checkoutNestedSubmodules {
67+
if($propertyvalue -ne $false){
68+
69+
$FilteredTaskProperties | Add-Member -NotePropertyName 'submodules' -NotePropertyValue 'recursive'
70+
71+
}
72+
}
73+
fetchDepth {
74+
if($propertyvalue -ne 0){
75+
$FilteredTaskProperties | Add-Member -NotePropertyName $propertyname -NotePropertyValue $propertyvalue
76+
}
77+
}
4178
Default {
4279
$FilteredTaskProperties | Add-Member -NotePropertyName $propertyname -NotePropertyValue $propertyvalue
4380
}

Source/Public/Get-AzDoAPIToolsDefinitionStepsAsYAMLPrepped.ps1

Lines changed: 117 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -27,52 +27,151 @@ function Get-AzDoAPIToolsDefinitionStepsAsYAMLPrepped {
2727
$jobcount = $jobs.count
2828
$definitionjobs = @()
2929
$retunreddefinitionjobs = [ordered]@{}
30-
$phaserefs = @{}
30+
[bool]$pipelinedemands = ($definition.PSobject.Properties.name.contains('demands'))
31+
3132

3233
foreach ($job in $jobs) {
34+
3335
$definitionsteps = [ordered]@{}
34-
$definitionjob = [ordered]@{}
35-
$steps = Convert-TaskStepsToYAMLSteps -InputArray $job -Projectname $projectname -profilename $profilename -inputtype $definitiontype -ExpandNestedTaskGroups:$ExpandNestedTaskGroups.isPresent
36-
37-
[bool]$custompool = ($job.target.PSobject.Properties.name.contains('queue'))
36+
$definitionjob = [ordered]@{}
37+
$demandstoadd = $null
38+
[array]$steps = @()
39+
[array]$stepsPlusStepproperties = @()
40+
$steppropertiestoadd = [ordered]@{}
41+
[bool]$custompool = ($job.target.PSobject.Properties.name.contains('queue'))
42+
[bool]$dependencies = ($job.PSobject.Properties.name.contains('dependencies'))
43+
[bool]$jobdemands = ($job.target.PSobject.Properties.name.contains('demands'))
44+
45+
#populating stepproperties
46+
47+
#first do checkout as it has to be the first task always
48+
49+
if($definition.repository.properties.skipSyncSource -eq 'true'){
50+
$steppropertiestoadd.add('checkout', 'none')
51+
}else {
52+
$steppropertiestoadd.add('checkout', 'self')
53+
}
3854

39-
[bool]$dependencies = ($job.PSobject.Properties.name.contains('dependencies'))
55+
#adding other properties
56+
$repooptions = Get-TaskProperties -InputTaskObject $definition.repository -propertiestoskip @('properties','id','type','name','url','defaultBranch')
57+
58+
$repooptions.PSobject.Properties | ForEach-Object{
59+
$steppropertiestoadd.add($_.name,$_.value)
60+
}
4061

41-
$phaserefs.Add($job.refName,$job.name)
62+
$repoproperties = Get-TaskProperties -InputTaskObject $definition.repository.properties -propertiestoskip @('labelSources','labelSourcesFormat','reportBuildStatus','cleanOptions','skipSyncSource')
63+
64+
$repoproperties.PSobject.Properties | ForEach-Object{
65+
if ($_.name -eq 'submodules') {
66+
$steppropertiestoadd.$($_.name) = $_.value
67+
}else{
68+
$steppropertiestoadd.add($_.name,$_.value)
69+
}
70+
}
71+
72+
if ($job.target.allowScriptsAuthAccessOption -eq 'true') {
4273

43-
if ($jobcount -gt 1 -or $custompool) {
74+
$steppropertiestoadd.add('persistCredentials', 'true')
75+
76+
}
77+
78+
if ($jobcount -gt 1 -or $custompool -or $pipelinedemands -or $jobdemands) {
4479

45-
$definitionjob.add('job',$job.name.replace(" ","_"))
80+
$definitionjob.add('job',$job.refName)
81+
### Adding displayname
82+
$definitionjob.add('displayName',$job.name)
83+
4684
### add job pool properties
47-
if ($custompool) {
85+
if ($custompool -and $job.target.type -eq 1) {
4886

4987
$poolToAdd = Get-AzDoAPIToolsAgentPool -poolURL $job.target.queue._links.self.href -AgentIdentifier $job.target.agentSpecification.identifier
50-
if ($pooltoAdd.count -ge 1) {
51-
$definitionjob.add('pool',$poolToAdd)
88+
89+
90+
[array]$stepsPlusStepproperties = $steppropertiestoadd
91+
92+
}elseif (!$custompool -and $job.target.type -eq 1) {
93+
94+
$poolToAdd = Get-AzDoAPIToolsAgentPool -PoolURL $definition.queue._links.self.href -agentidentifier $definition.process.target.agentSpecification.identifier
95+
96+
[array]$stepsPlusStepproperties = $steppropertiestoadd
97+
}
98+
elseif($job.target.type -eq 2){
99+
100+
$poolToAdd = 'server'
101+
102+
[array]$stepsPlusStepproperties = @()
103+
}
104+
105+
if ($pooltoAdd.count -ge 1) {
106+
$definitionjob.add('pool',$poolToAdd)
107+
}
108+
109+
### Adding job demands
110+
if ($jobdemands) {
111+
$demandstoadd += $job.target.demands
112+
}
113+
114+
if ($pipelinedemands) {
115+
$demandstoadd += $definition.demands
116+
}
117+
118+
if ($demandstoadd.count -ge 1 -and $job.target.type -eq 1) {
119+
if($definitionjob.Contains('pool')){
120+
$definitionjob.pool.add('demands',$demandstoadd)
121+
}else{
122+
Write-Error "No Pool construct found to add demands to."
52123
}
124+
125+
}
126+
### Adding Job Properties
127+
128+
#pipeline specific timeouts
129+
if($definition.jobTimeoutInMinutes -ne 60){
130+
$definitionjob.add('timeoutInMinutes',$definition.jobTimeoutInMinutes)
131+
53132
}
54133

134+
if($definition.jobCancelTimeoutInMinutes -ne 5){
135+
$definitionjob.add('cancelTimeoutInMinutes',$definition.jobCancelTimeoutInMinutes)
136+
}
137+
138+
#job specific properties
55139
$jobproperties = Get-TaskProperties -InputTaskObject $job -propertiestoskip @('steps','target','name','refname','jobAuthorizationScope','dependencies')
56140

57141
$jobproperties.PSObject.Properties | ForEach-Object{
58-
$definitionjob.add($_.name,$_.value)
142+
if ($_.name -eq 'timeoutInMinutes' -and ($definitionjob.timeoutInMinutes)) {
143+
$definitionjob.timeoutInMinutes = $_.value
144+
145+
}elseif ($_.name -eq 'cancelTimeoutInMinutes' -and ($definitionjob.cancelTimeoutInMinutes)) {
146+
$definitionjob.cancelTimeoutInMinutes = $_.value
147+
148+
}else{
149+
$definitionjob.add($_.name,$_.value)
150+
}
151+
59152
}
60153

61154
#add section for dependancies
62155
if ($dependencies) {
63-
$dependancy = $phaserefs.$($job.dependencies.scope)
64-
$definitionjob.add('dependsOn',$dependancy.replace(" ","_"))
156+
$definitionjob.add('dependsOn',$job.dependencies.scope)
65157
}
158+
159+
#populating jobs/steps
160+
161+
[array]$steps = Convert-TaskStepsToYAMLSteps -InputArray $job -Projectname $projectname -profilename $profilename -inputtype $definitiontype -ExpandNestedTaskGroups:$ExpandNestedTaskGroups.isPresent
66162

67-
$definitionjob.add('steps',$steps)
163+
[array]$stepsPlusStepproperties += $steps
164+
165+
$definitionjob.add('steps',$stepsPlusStepproperties)
68166

69167
$definitionjobs += $definitionjob
168+
70169
}else{
71-
$definitionsteps.add('steps',$steps)
170+
$definitionsteps.add('steps',$stepsPlusStepproperties)
72171
}
73172
}
74173

75-
if ($jobcount -gt 1 -or $custompool) {
174+
if ($jobcount -gt 1 -or $custompool -or $pipelinedemands -or $jobdemands) {
76175
$retunreddefinitionjobs.add('jobs',$definitionjobs)
77176

78177
return $retunreddefinitionjobs

Source/Public/Get-AzDoAPIToolsDefinitionTriggersAsYAMLPrepped.ps1

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,12 @@ function Get-AzDoAPIToolsDefinitionTriggersAsYAMLPrepped {
1717

1818
if ($citriggers) {
1919
$triggers = [ordered]@{}
20+
21+
if ($citriggers.batchChanges -eq 'true') {
22+
23+
$triggers.add('batch',$citriggers.batchChanges)
24+
}
25+
2026
if($citriggers.branchFilters){
2127
$branchtriggers = Get-DefinitionInputIncludeExclude -inputs $citriggers.branchFilters
2228

@@ -33,12 +39,6 @@ function Get-AzDoAPIToolsDefinitionTriggersAsYAMLPrepped {
3339
}
3440

3541
}
36-
37-
38-
if ($citriggers.batchChanges -eq 'true') {
39-
40-
$triggers.add('batch',$citriggers.batchChanges)
41-
}
4242

4343
if ($triggers.Count -ge 1) {
4444
$returnedtriggerobject = @{

0 commit comments

Comments
 (0)