Unfortunately the latest v2 version of the arduino/setup-task action used to install the Task task runner tool in the
runner machine has a dependency on a higher version of glibc than is provided by the Linux container. For this reason,
the workflow is configured to use arduino/setup-task@v1 for the Linux build job.
We will receive pull requests from Dependabot offering to update this outdated action dependency at each subsequent
major version release of the action (which are not terribly frequent). We must decline the bump of the action in that
specific step, but we can accept the bumps of all other usages of the action in the workflows. Dependabot remembers when
you decline a bump so this should not be too bothersome.
A Docker container is used to produce the Linux build of Arduino IDE.
The dependencies of the build are pinned to a specific version in the Dockerfile in order to ensure a stable environment
in the images.
One such dependency is Git. The version of Git available from the package repository of the Ubuntu 18.04 distro used for
the image is too outdated to be used for this purpose. For this reason, a PPA is used as the source for the Git package.
Unfortunately the maintainers of the PPA remove the previous package every time they add a package for a newer version
of Git. This breaks the version pinned installation of the Git package:
5.515 E: Version '1:2.42.0-0ppa1~ubuntu18.04.1' for 'git' was not found
For this reason it is necessary to unpin Git in the Dockerfile. This will cause whichever version is available from the
PPA to be installed each time the image is built. Although not ideal for this application, that shouldn't cause any
problems in practice since Git has quite a stable and carefully maintained interface.
Unfortunately the latest v5 version of the actions/setup-go action used to set up the Go programming language in the
runner machine has a dependency on a higher version of glibc than is provided by the Linux container. For this reason,
the workflow is configured to use actions/setup-go@v4 for the Linux build job. We will receive pull requests from
Dependabot offering to update this outdated action dependency for at each subsequent major version release of the action
(which are not terribly frequent). We must decline the bump of the action in that specific step, but accept the bumps of
all other usages of the action in the workflows. Dependabot remembers when you decline a bump so this should not be too
bothersome.
The "Arduino IDE" GitHub Actions workflow uses a Docker container for the build job of certain target, while running others directly in the
runner environment.
Due to differences between these two environments, some steps must run only when a container is used by the job, and others only when the job is running in the runner environment. This is done by a conditional on the job matrix data regarding the container. The container value is set to null for the jobs that run in the runner environment, while containing a full container configuration mapping for the jobs that run in a container.
Previously the conditional unnecessarily used the value of the image key of the container object specifically. The comparison
can be done against the container value itself. Doing this will make it a little easier to understand the workflow code, since the conditional more clearly reflects the matrix (where `container` is set to `null` instead of `container.image`).
Python 3.12.x is incompatible with the current version of node-gyp (9.4.0). For this reason, it is necessary to
configure the "GitHub Actions" workflow to use the compatible Python 3.11.x until the next release of node-gyp (which
should contain a fix for the breakage) is made.
In addition to the builds, when a nightly or production release is published, a "channel update info file" is also
uploaded to Amazon S3 by the "Arduino IDE" GitHub Actions workflow. The IDE checks the channel file on the server to get
information about available updates.
Previously the Linux production release builds were being remade manually due to the ones produced by GitHub Actions not
being compatible with older distro versions due to a dynamically linked dependency. For this reason, a step was
temporarily added to the workflow to cause it to not upload the Linux channel file in order to avoid Linux users from
receiving an update offer before the limited compatibility automated build had been replaced with the manually produced
build.
The automated build system has been adjusted to produce Linux builds with the intended range of compatibility, but the
step that deleted the channel file was not removed at that time. The obviated step is hereby removed in order to allow
complete releases to be published by the workflow.
The "Arduino IDE" GitHub Actions workflow is used to generate several distinct types of builds:
- Tester builds of commits
- Nightly builds
- Release builds
Different actions must be performed depending on which type of build is being produced. The workflow uses dedicated jobs
for publishing the nightly builds and for publishing the release builds. Those jobs are configured to run only when
certain criteria are met.
One such criteria is that the merge-channel-files job ran as expected. There are four possible result types of a job,
which should be handled as follows:
| Result | Run dependent job? |
| -------- | ------------------ |
| success | Yes |
| failure | No |
| canceled | No |
| skipped | Yes |
GitHub Actions automatically takes the desired action regarding whether the dependent job should run for the first three
result types, but that is not the case for the "skipped" result. The merge-channel-files job dependency is skipped when
a channel file merge is not needed and so this is not cause to cancel the build publishing.
The only way to make a dependent job run when the dependency was skipped is to add the `always()` expression to the job
conditional. This goes too far in the other direction by causing the job to run even when the dependency failed or was
canceled. So it is necessary to also add logic for each of the dependency job result types to the conditional, which
makes it quite complex when combined with the logic for the other criteria of the job.
In order to reduce the amount of complexity of the conditionals of the dependent jobs, a job was interposed in the
dependency chain, which was intended to act simply as a container for the logic about the merge-channel-files job
result. Unfortunately it turns out that even if the direct dependency job's result was success, if any ancestor in the
dependency chain was skipped, GitHub Actions still skips all dependent jobs without an `always()` expression in their
conditional, meaning the intermediate job was pointless. This caused the build publishing jobs to be skipped under the
conditions where they should have ran.
The pointless intermediate job is hereby removed, an `always()` expression added to the conditionals of the dependent
jobs, and the full logic for how to handle each dependent job result type added there as well.
Background
----------
The Linux build of Arduino IDE is dynamically linked against the libstdc++ and glibc shared libraries. This results in
it having a dependency on the version of the libraries that happens to be present in the environment it is built in.
Although newer versions of the shared libraries are compatible with executables linked against an older version, the
reverse is not true. This means that building Arduino IDE on a Linux machine with a recent distro version installed
causes the IDE to error on startup for users who have a distro with older versions of the dependencies. For example:
```
Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/per/Downloads/arduino-ide_nightly-20231006_Linux_64bit/resources/app/lib/backend/native/nsfw.node)
```
or:
```
Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/per/Downloads/arduino-ide_2.0.5-snapshot-90b1f67_Linux_64bit/resources/app/node_modules/nsfw/build/Release/nsfw.node)
```
We were originally able to achieve our targeted range of Linux distro compatibility by running the Linux build in the
ubuntu-18.04 GitHub Actions hosted runner machine. Unfortunately GitHub stopped offering that runner machine. This meant
we were forced to update to using the ubuntu-20.04 runner machine instead, which caused the loss of compatibility of
the automatically generated builds with previously supported distro versions (e.g., Ubuntu 18.04). Since that time, the
release builds have been produced manually, which is inefficient and prone to human error.
Update
------
The identified solution to restoring fully automated builds with the target range of Linux distro compatibility is to
run the build in a Docker container that provides the suitable environment. This means a combination of an older distro
version with the modern versions of the development tool dependencies of the build.
Such a combination was achieved by creating a bespoke image based on the ubuntu:18.04 image. The Dockerfile is hosted in
the Arduino IDE repository in order to allow it to be maintained in parallel with the code and infrastructure.
Image Publishing
----------------
A "Push Container Images" GitHub Actions continuous delivery workflow is added to push updated images to the GitHub
Container registry when a commit that modifies relevant files is pushed to the main branch.
This means the image does not have formally versioned tags and the IDE build uses the container that results from the
configuration at the tip of the main branch at the time of the build. I think that is a reasonable approach in this use
case where the image is targeted to a single application rather than intended to be used by multiple projects.
Container Validation
--------------------
The build workflow is configured to trigger on completion of that push workflow in order to provide validation of the IDE build using the
updated container as well as the resulting tester builds of the IDE.
A "Check Containers" GitHub Actions continuous integration workflow is added to provide basic validation for changes to
the Dockerfile. It will automatically build the image and run the container on any push or pull request that modifies
relevant files.
Container Workflow Design
-------------------------
With the goal of reusability, the image data is contained in a job matrix in the workflow to allow them to accommodate
any number of arbitrary images.
Build Workflow Configuration
----------------------------
A container property is added to the build job matrix data. If the container.image property is set to null, GitHub
Actions will run the job directly in the runner environment instead of in a container. This allows us to produce the
builds using either a container or a bare runner machine as is appropriate for each target.
Unfortunately the latest v4 version of the actions/checkout action used to checkout the repository into the job
environment has a dependency on a higher version of glibc than is provided by the Linux container. For this reason, the
workflow is configured to use actions/checkout@v3 for the Linux build job. We will likely receive pull requests from
Dependabot offering to update this outdated action dependency for the v4 and at each subsequent major version release of
the action (which are not terribly frequent). We must decline the bump of the action in that specific step, but accept
the bumps of all other usages of the action in the workflows. Dependabot remembers when you decline a bump so this
should not be too bothersome.
- fix(test): integration tests are more resilient.
- run the Create integration tests with other slow tests,
- queued `PUT`/`DELETE` requests to make the test timeout happy,
- reduced the `/sketches/search` offset to 1/5th and
- remove Create API logging.
- fix(linter): ignore `lib` folder. Remove obsolete `.node_modules`
pattern.
- feat(ci): enable Create API integration tests.
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>
Co-authored-by: per1234 <accounts@perglass.com>
The "Arduino IDE" GitHub Actions workflow uploads the nightly and release builds to Amazon S3, from which they are
downloaded by the auto-update as well as directly by users via the links on the "Software" page of arduino.cc.
The workflow can also be useful in forks. Either by those who want to test contributions staged in their fork prior to
submitting a PR to the parent repo, or by those maintaining a hard fork of the project. Even though these forks wouldn't
(and couldn't due to lack of access to the encrypted credential secrets only available to the workflow when ran in a
trusted context in Arduino's repo)credentials stored in Arduino's repo) use the S3 upload component of the workflow,
they may still find it valuable for continuous integration as well as continuous deployment via the tester builds and
release builds the workflow also publishes to the GitHub repository it runs in. For this reason, the workflow contains
code to determine whether it should attempt the S3 uploads.
Previously the repository name was used as the criteria in that code. The project specificity of that approach makes the
workflow less easily reusable. A more generally applicable criterion is whether the encrypted credential certificate is
defined.
The new criterion allows the workflow to be used in any repository where the administrator has created an encrypted
secret containing their AWS credentials. That might be other projects owned by Arduino, or even 3rd party projects where
the owners want to take a similar build publishing approach using their own AWS account.
The "Arduino IDE" GitHub Actions workflow uploads the nightly and release builds to Amazon S3, from which they are
downloaded by the auto-update as well as directly by users via the links on the "Software" page of arduino.cc.
The workflow can also be useful in forks. Either by those who want to test contributions staged in their fork prior to
submitting a PR to the parent repo, or by those maintaining a hard fork of the project. Even though these forks wouldn't
(and couldn't due to lack of access to the encrypted credential secrets only available to the workflow when ran in a
trusted context in Arduino's repo)credentials stored in Arduino's repo) use the S3 upload component of the workflow,
they may still find it valuable for continuous integration as well as continuous deployment via the tester builds and
release builds the workflow also publishes to the GitHub repository it runs in. For this reason, the workflow contains
code to determine whether it should attempt the S3 uploads. Previously that code was duplicated in both the nightly and
release publishing jobs of the workflow. Since the workflow already contains a job specifically for the purpose of
determining the characteristics of the build being performed and making that information available from single source
for use throughout the rest of the workflow, it makes sense to also move the S3 upload determination code to that job.
On every release tag, and manual trigger when the "Include builds on non-free runners" checkbox is checked, make the
Arduino IDE build for native Apple Silicon host in addition to the builds that are always generated by the "Arduino IDE"
GitHub Actions workflow.
Previously, the build workflow only produced a build for x86-64 (AKA "Intel") macOS hosts. Although it is possible to
use those builds on Apple Silicon machines via the Rosetta 2 translation software, the performance is significantly
inferior to a native build so we must also provide Apple Silicon native builds.
Previously the Apple Silicon builds were produced manually. The reason for using that inefficient and error-prone
approach instead of the automated continuous deployment system used for other builds was that GitHub did not provide the
necessary Apple Silicon runner machines and Arduino was not capable of setting up such self-hosted machines in a manner
that would make them feasible for the project maintainers to use. GitHub hosted Apple Silicon runner machines are now
available so we can add the target to the build workflow.
GitHub gives unlimited use of the basic runner machines for workflow runs in public repositories. However, the macOS ARM
architecture is only provided in runner machines which are classified as "larger runner". Use of these runners is
charged on a per-minute basis, without any of the free allowances GitHub provides for the normal runners. In order to
avoid unnecessary expenditures, native Apple Silicon builds must be generated only when there is compelling reason to do
so. Such a build is needed for every release, so the workflow is configured to always generate the builds when triggered
by a tag. In addition to releases, Apple Silicon tester builds for pull requests that might have special implications
for this target. For this reason, the workflow is configured to allow Apple Silicon builds to be triggered manually by a
repository maintainer.
The workflow uses a job matrix to run the build for each target on the appropriate runner machine in parallel, using the
universally applicable workflow code for all jobs. It uses another job matrix to generate individual workflow artifacts
for the tester builds of each target. Previously it was possible to always use the same matrix configurations for all
workflow runs. With the addition of the selectively run macOS ARM job, it is now necessary to generate these matrixes on
the fly.
The electron-updater package used by Arduino IDE's auto-update capability uses a data file (known as the "channel update
info file") to check for the availability of updates. A single "channel update info file" is used for the data of the
macOS x86 and ARM builds. Since a separate job is used to produce each of those builds, this means the "channel update
info file" produced by each of the macOS build jobs must be merged into a single file.
The "Arduino IDE" GitHub Actions workflow is used to generate several distinct types of builds:
- Tester builds of commits
- Nightly builds
- Release builds
Different actions must be performed depending on which type of build is being produced. The workflow contains code that
uses various criteria to determine the build type.
Previously that code was duplicated in multiple places:
- The packaging job
- The changelog generation job
- The nightly build publishing job
- The release publishing job
This duplication is avoided by moving the code to a dedicated job that makes the build type information available to all
subsequent jobs via outputs.
The "Arduino IDE" GitHub Actions workflow uses a job matrix to make the builds for each host target in parallel.
The same steps are used for each job, but some configuration adjustments must be made on a per-target basis. This is
done through various attributes in the matrix configuration.
Previously the `os` attribute was used for two distinct things:
- The machine identifier of the GitHub Actions runner machine of the job.
- The differentiator in the human-targeted job name.
The attribute name "os" (for "operating system") was misleading because runners are differentiated by more than only the
operating system on the machine.
The use of a machine identifier as a differentiator in the human-targeted job name was a bad idea because these
identifiers would only effectively communicate the nature of a build to humans who are quite knowledgeable about the
GitHub Actions workflow syntax.
The impact of these poor decisions has not been too severe previously due to there only being a single job for each
operating system. However, there is a need for multiple jobs per operating system in order to support multiple host
architectures (e.g., macOS x86 and ARM).
The solution is to:
- Use an appropriate name for the runner identifier attribute.
- Use a dedicated attribute for the human friendly job name differentiator.
The project was recently switched to using a "trunk-based" development strategy. This necessitated some adjustments to
the configuration of the GitHub Actions workflows in order to ensure the CI system could be used to effectively validate
the project at the state staged for release in the release branch.
A `run-determination` job was added to the workflow. This job determines whether the conditions under which the workflow
was triggered indicate that the rest of the jobs in the workflow should be run.
A validation workflow should run fully under any of the following conditions:
- The trigger event was something other than a branch creation
- The trigger event was a release branch creation
Since the project is fully validated prior to a release, running the workflow when triggered by a release tag is
pointless (and even harmful in some specific standardized workflows not currently used in this repository), so (even if
for no other reason than efficiency) verification workflows are configured to not run under these conditions.
The standardized Arduino tooling workflows follow a modular design where each workflow has a narrow scope of purpose.
That path was not taken by those who set up the infrastructure for this repository. They instead created a single
massive monolithic "Arduino IDE" workflow that performs many unrelated operations under various conditions. This workflow generates
releases in addition to doing validation. Even though it is pointless to run the workflow's validation operations when
the workflow is triggered by a release tag, it is essential to run it for the release generation.
Previously, the code used in the "Arduino IDE" workflow's `run-determination` job was configured as appropriate for a
verification workflow. This meant that a release would not be generated on push of a release tag as intended. The bug is
fixed by adjusting the code to do a full run of the workflow when triggered by a release tag.
- update Theia to `1.39.0`,
- remove the application packager and fix the security vulnerabilities,
- bundle the backed application with `webpack`, and
- enhance the developer docs.
Co-authored-by: Akos Kitta <a.kitta@arduino.cc>
Co-authored-by: per1234 <accounts@perglass.com>
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>
Previously, releases were always made from a point in the revision history of the `main` branch (typically the tip of
the branch at the time of the release). Although the simplicity of this approach is nice, it can be limiting in some
cases. For this reason, the project is switching to using a "trunk-based" development strategy, as described here:
https://trunkbaseddevelopment.com/
This approach allows making releases at any time that consist of the arbitrary subset of revisions suitable for shipping
to the users at that time. The commits that should be included in the release are cherry-picked to a release branch and
the tag created on that branch.
This means that:
- PRs can be merged to the `main` branch as soon as they have passed review rather than having to postpone the merge of
changes that are not ready to be included in the next release.
- Releases don't need to be postponed if the prior revision history on the `main` branch contains changes that are not
ready to be included in the release.
The documented release procedure must be adjusted to reflect the new development strategy.
CI System Adjustments
---------------------
The status of the GitHub Actions workflows should be evaluated before making a release. However, this is not so simple as
checking the status of the commit at the tip of the release branch. The reason is that, for the sake of efficiency, the
workflows are configured to run only when the processes are relevant to the trigger event (e.g., no need to run unit
tests for a change to the readme).
In the case of the default branch, you can simply set the workflow runs filter to that branch and then check the result
of the latest run of each workflow of interest. However, that was not possible to do with the release branch since it
might be that the workflow was never run in that branch. The status of the latest run of the workflow in the default
branch might not match the status for the release branch if the release branch does not contain the full history.
For this reason, it will be helpful to trigger all relevant workflows on the creation of a release branch. This will
ensure that each of those workflows will always have at least one run in the release branch. Subsequent commits pushed to
the branch can run based on their usual trigger filters and the status of the latest run of each workflow in the branch
will provide an accurate indication of the state of that branch.
Branches are created for purposes other than releases, most notably feature branches to stage work for a pull request.
Due to the comprehensive nature of this project's CI system, it would not be convenient or efficient to fully run all CI
workflows on the creation of every feature branch.
Unfortunately, GitHub Actions does not support filters on the `create` event of branch creation like it does for the
`push` and `pull_request` events. There is support for a `branches` filter of the `push` event, but that filter is an
"AND" to the `paths` filter while this application requires an "OR". For this reason, the workflows must be triggered by
the creation of any branch. The unwanted job runs are prevented by adding a `run-determination` job with the branch
filter handled by Bash commands. The other jobs of the workflow use this `run-determination` job as a dependency, only
running when it indicates they should via a job output. Because this minimal `run-determination` job runs very quickly,
it is roughly equivalent to the workflow having been skipped entirely for non-release branch creations.
- Updated `@theia/*` to `1.37.0`.
- Fixed all `yarn audit` security vulnerabilities.
- Updated to `electron@23.2.4`:
- `contextIsolation` is `true`,
- `nodeIntegration` is `false`, and the
- `webpack` target is moved from `electron-renderer` to `web`.
- Updated to `typescript@4.9.3`.
- Updated the `eslint` plugins.
- Added the new `Light High Contrast` theme to the IDE2.
- High contrast themes use Theia APIs for style adjustments.
- Support for ESM modules: `"moduleResolution": "node16"`.
- Node.js >= 16.14 is required.
- VISX langage packs were bumped to `1.70.0`.
- Removed undesired editor context menu items. (Closes#1394)
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>
Arduino IDE offers an update to the user when a newer version is available. The availability of an update is determined
by comparing the user's IDE version against data file ("channel update info file") stored on Arduino's download server.
These "channel update info files" are automatically generated by the build workflow.
Previously the release process was fully automated, including the upload of the "channel update info files" to the
server.
As a temporary workaround for limitations of the GitHub Actions runner machines used to produce the automated builds,
some release builds are now produced manually:
- Linux build (because the Ubuntu 18.04 runner was shut down and newer runner versions produce builds incompatible with
older Linux versions)
- macOS Apple Silicon build (because GitHub hosted Apple Silicon runners are not available)
The automatic upload of the "channel update info files" produced by the build workflow is problematic because if users
receive update offers before the "channel update info files" are updated for the manually produced builds, they can
receive an update to a different build than intended:
- Users of older Linux versions would update to a build that won't start on their machine
- macOS Apple Silicon users would update to macOS x86 build that is less performant on their machine
For this reason, the build workflow is adjusted to no longer upload the Linux and macOS "channel update info files" to
the download server on release. These files will now be manually uploaded after they have been updated to provide the
manually produced builds.
This workaround will be reverted once a fully automated release system is regained.
- updated to `electron-builder@23.6.0`
- set `CSC_FOR_PULL_REQUEST` env to run notarization for a PR build.
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>
The "Check Certificates" GitHub Actions workflow uses OpenSSL to check for problems with the project's signing
certificates.
Certificates exported to PKS#12 archive files using older tools may have been encrypted using the "RC2-40-CBC"
algorithm.
Due to the availability of more secure modern alternatives, default support for RC2-40-CBC encryption was dropped in
OpenSSL 3.x.
The macOS signing certificate uses this RC2-40-CBC encryption.
The "Check Certificates" GitHub Actions workflow runs on the `ubuntu-latest` runner. Previously, this runner used Ubuntu
20.04. This has now changed to Ubuntu 22.04. With the operating system update came an OpenSSL update from 1.1.1f to
3.0.2. This caused the workflow runs to fail on the macOS certificate job:
Error outputting keys and certificates
80FBB0C5087F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:../crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()
Even though no longer done by default, OpenSSL still supports RC2-40-CBC encryption via its "legacy" provider. So
compatibility with the certificate is restored by adding the `-legacy` flag to the `openssl pkcs12` commands.
The "Arduino IDE" GitHub Actions workflow generates a changelog from the commits since the last tag.
This changelog is published in multiple ways:
- Printed to workflow run logs
- Uploaded to Arduino's download server (mostly useful for the nightly builds)
- Initial version of release notes
For the last, the changelog text must be passed from the dedicated changelog generation workflow step to the release
step. This is done via workflow job output.
At the time the system was set up, outputs for workflow `run` steps were set using the `set-output` workflow command.
That "workflow command" system was later determined by GitHub to have potential security vulnerabilities, so it was
replaced with a `GITHUB_OUTPUT` environment file.
The "Arduino IDE" workflow was migrated to the new "environment file" approach. It was later discovered that there was
an undocumented breaking change in the method for handling multi-line strings in workflow step outputs between the old
"workflow command" system and the new "environment file". This resulted in the initial release notes having an incorrect
format. For example, what would previously have been formatted like this:
- Updated translation files (#1606) [23c7f5f]
- Use 0.29.0 CLI in IDE2 (#1683) [f1144ef]
Was now formatted like this:
- Updated translation files (#1606) [23c7f5f]%0A - Use 0.29.0 CLI in IDE2 (#1683) [f1144ef]%0A
The solution is to remove the commands that did the escaping of the changelog text in a manner that is no longer
supported and replace them with a "here document"-style format.
A random number is used as the "delimiter" (limit string) per the security recommendations in the official GitHub
documentation. Note that even though the multiline strings handling documentation was placed under the environment
variable section, it also applies to setting outputs.
