A Docker container is used to produce the Linux build of Arduino IDE.
The dependencies of the build are pinned to a specific version in the Dockerfile in order to ensure a stable environment
in the images.
One such dependency is Git. The version of Git available from the package repository of the Ubuntu 18.04 distro used for
the image is too outdated to be used for this purpose. For this reason, a PPA is used as the source for the Git package.
Unfortunately the maintainers of the PPA remove the previous package every time they add a package for a newer version
of Git. This breaks the version pinned installation of the Git package:
5.515 E: Version '1:2.42.0-0ppa1~ubuntu18.04.1' for 'git' was not found
For this reason it is necessary to unpin Git in the Dockerfile. This will cause whichever version is available from the
PPA to be installed each time the image is built. Although not ideal for this application, that shouldn't cause any
problems in practice since Git has quite a stable and carefully maintained interface.
Unfortunately the latest v5 version of the actions/setup-go action used to set up the Go programming language in the
runner machine has a dependency on a higher version of glibc than is provided by the Linux container. For this reason,
the workflow is configured to use actions/setup-go@v4 for the Linux build job. We will receive pull requests from
Dependabot offering to update this outdated action dependency for at each subsequent major version release of the action
(which are not terribly frequent). We must decline the bump of the action in that specific step, but accept the bumps of
all other usages of the action in the workflows. Dependabot remembers when you decline a bump so this should not be too
bothersome.
The "Arduino IDE" GitHub Actions workflow uses a Docker container for the build job of certain target, while running others directly in the
runner environment.
Due to differences between these two environments, some steps must run only when a container is used by the job, and others only when the job is running in the runner environment. This is done by a conditional on the job matrix data regarding the container. The container value is set to null for the jobs that run in the runner environment, while containing a full container configuration mapping for the jobs that run in a container.
Previously the conditional unnecessarily used the value of the image key of the container object specifically. The comparison
can be done against the container value itself. Doing this will make it a little easier to understand the workflow code, since the conditional more clearly reflects the matrix (where `container` is set to `null` instead of `container.image`).
Python 3.12.x is incompatible with the current version of node-gyp (9.4.0). For this reason, it is necessary to
configure the "GitHub Actions" workflow to use the compatible Python 3.11.x until the next release of node-gyp (which
should contain a fix for the breakage) is made.
In addition to the builds, when a nightly or production release is published, a "channel update info file" is also
uploaded to Amazon S3 by the "Arduino IDE" GitHub Actions workflow. The IDE checks the channel file on the server to get
information about available updates.
Previously the Linux production release builds were being remade manually due to the ones produced by GitHub Actions not
being compatible with older distro versions due to a dynamically linked dependency. For this reason, a step was
temporarily added to the workflow to cause it to not upload the Linux channel file in order to avoid Linux users from
receiving an update offer before the limited compatibility automated build had been replaced with the manually produced
build.
The automated build system has been adjusted to produce Linux builds with the intended range of compatibility, but the
step that deleted the channel file was not removed at that time. The obviated step is hereby removed in order to allow
complete releases to be published by the workflow.
The "Arduino IDE" GitHub Actions workflow is used to generate several distinct types of builds:
- Tester builds of commits
- Nightly builds
- Release builds
Different actions must be performed depending on which type of build is being produced. The workflow uses dedicated jobs
for publishing the nightly builds and for publishing the release builds. Those jobs are configured to run only when
certain criteria are met.
One such criteria is that the merge-channel-files job ran as expected. There are four possible result types of a job,
which should be handled as follows:
| Result | Run dependent job? |
| -------- | ------------------ |
| success | Yes |
| failure | No |
| canceled | No |
| skipped | Yes |
GitHub Actions automatically takes the desired action regarding whether the dependent job should run for the first three
result types, but that is not the case for the "skipped" result. The merge-channel-files job dependency is skipped when
a channel file merge is not needed and so this is not cause to cancel the build publishing.
The only way to make a dependent job run when the dependency was skipped is to add the `always()` expression to the job
conditional. This goes too far in the other direction by causing the job to run even when the dependency failed or was
canceled. So it is necessary to also add logic for each of the dependency job result types to the conditional, which
makes it quite complex when combined with the logic for the other criteria of the job.
In order to reduce the amount of complexity of the conditionals of the dependent jobs, a job was interposed in the
dependency chain, which was intended to act simply as a container for the logic about the merge-channel-files job
result. Unfortunately it turns out that even if the direct dependency job's result was success, if any ancestor in the
dependency chain was skipped, GitHub Actions still skips all dependent jobs without an `always()` expression in their
conditional, meaning the intermediate job was pointless. This caused the build publishing jobs to be skipped under the
conditions where they should have ran.
The pointless intermediate job is hereby removed, an `always()` expression added to the conditionals of the dependent
jobs, and the full logic for how to handle each dependent job result type added there as well.
To fix all security vulnerabilities detected by `Dependabot`.
- remove `shelljs`. replace with `fs` and `console`.
- remove `uuid`. replace with `@phosphor/coreutils`.
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>
Background
----------
The Linux build of Arduino IDE is dynamically linked against the libstdc++ and glibc shared libraries. This results in
it having a dependency on the version of the libraries that happens to be present in the environment it is built in.
Although newer versions of the shared libraries are compatible with executables linked against an older version, the
reverse is not true. This means that building Arduino IDE on a Linux machine with a recent distro version installed
causes the IDE to error on startup for users who have a distro with older versions of the dependencies. For example:
```
Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/per/Downloads/arduino-ide_nightly-20231006_Linux_64bit/resources/app/lib/backend/native/nsfw.node)
```
or:
```
Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/per/Downloads/arduino-ide_2.0.5-snapshot-90b1f67_Linux_64bit/resources/app/node_modules/nsfw/build/Release/nsfw.node)
```
We were originally able to achieve our targeted range of Linux distro compatibility by running the Linux build in the
ubuntu-18.04 GitHub Actions hosted runner machine. Unfortunately GitHub stopped offering that runner machine. This meant
we were forced to update to using the ubuntu-20.04 runner machine instead, which caused the loss of compatibility of
the automatically generated builds with previously supported distro versions (e.g., Ubuntu 18.04). Since that time, the
release builds have been produced manually, which is inefficient and prone to human error.
Update
------
The identified solution to restoring fully automated builds with the target range of Linux distro compatibility is to
run the build in a Docker container that provides the suitable environment. This means a combination of an older distro
version with the modern versions of the development tool dependencies of the build.
Such a combination was achieved by creating a bespoke image based on the ubuntu:18.04 image. The Dockerfile is hosted in
the Arduino IDE repository in order to allow it to be maintained in parallel with the code and infrastructure.
Image Publishing
----------------
A "Push Container Images" GitHub Actions continuous delivery workflow is added to push updated images to the GitHub
Container registry when a commit that modifies relevant files is pushed to the main branch.
This means the image does not have formally versioned tags and the IDE build uses the container that results from the
configuration at the tip of the main branch at the time of the build. I think that is a reasonable approach in this use
case where the image is targeted to a single application rather than intended to be used by multiple projects.
Container Validation
--------------------
The build workflow is configured to trigger on completion of that push workflow in order to provide validation of the IDE build using the
updated container as well as the resulting tester builds of the IDE.
A "Check Containers" GitHub Actions continuous integration workflow is added to provide basic validation for changes to
the Dockerfile. It will automatically build the image and run the container on any push or pull request that modifies
relevant files.
Container Workflow Design
-------------------------
With the goal of reusability, the image data is contained in a job matrix in the workflow to allow them to accommodate
any number of arbitrary images.
Build Workflow Configuration
----------------------------
A container property is added to the build job matrix data. If the container.image property is set to null, GitHub
Actions will run the job directly in the runner environment instead of in a container. This allows us to produce the
builds using either a container or a bare runner machine as is appropriate for each target.
Unfortunately the latest v4 version of the actions/checkout action used to checkout the repository into the job
environment has a dependency on a higher version of glibc than is provided by the Linux container. For this reason, the
workflow is configured to use actions/checkout@v3 for the Linux build job. We will likely receive pull requests from
Dependabot offering to update this outdated action dependency for the v4 and at each subsequent major version release of
the action (which are not terribly frequent). We must decline the bump of the action in that specific step, but accept
the bumps of all other usages of the action in the workflows. Dependabot remembers when you decline a bump so this
should not be too bothersome.
- fix(test): integration tests are more resilient.
- run the Create integration tests with other slow tests,
- queued `PUT`/`DELETE` requests to make the test timeout happy,
- reduced the `/sketches/search` offset to 1/5th and
- remove Create API logging.
- fix(linter): ignore `lib` folder. Remove obsolete `.node_modules`
pattern.
- feat(ci): enable Create API integration tests.
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>
Co-authored-by: per1234 <accounts@perglass.com>
The "Arduino IDE" GitHub Actions workflow uploads the nightly and release builds to Amazon S3, from which they are
downloaded by the auto-update as well as directly by users via the links on the "Software" page of arduino.cc.
The workflow can also be useful in forks. Either by those who want to test contributions staged in their fork prior to
submitting a PR to the parent repo, or by those maintaining a hard fork of the project. Even though these forks wouldn't
(and couldn't due to lack of access to the encrypted credential secrets only available to the workflow when ran in a
trusted context in Arduino's repo)credentials stored in Arduino's repo) use the S3 upload component of the workflow,
they may still find it valuable for continuous integration as well as continuous deployment via the tester builds and
release builds the workflow also publishes to the GitHub repository it runs in. For this reason, the workflow contains
code to determine whether it should attempt the S3 uploads.
Previously the repository name was used as the criteria in that code. The project specificity of that approach makes the
workflow less easily reusable. A more generally applicable criterion is whether the encrypted credential certificate is
defined.
The new criterion allows the workflow to be used in any repository where the administrator has created an encrypted
secret containing their AWS credentials. That might be other projects owned by Arduino, or even 3rd party projects where
the owners want to take a similar build publishing approach using their own AWS account.
The "Arduino IDE" GitHub Actions workflow uploads the nightly and release builds to Amazon S3, from which they are
downloaded by the auto-update as well as directly by users via the links on the "Software" page of arduino.cc.
The workflow can also be useful in forks. Either by those who want to test contributions staged in their fork prior to
submitting a PR to the parent repo, or by those maintaining a hard fork of the project. Even though these forks wouldn't
(and couldn't due to lack of access to the encrypted credential secrets only available to the workflow when ran in a
trusted context in Arduino's repo)credentials stored in Arduino's repo) use the S3 upload component of the workflow,
they may still find it valuable for continuous integration as well as continuous deployment via the tester builds and
release builds the workflow also publishes to the GitHub repository it runs in. For this reason, the workflow contains
code to determine whether it should attempt the S3 uploads. Previously that code was duplicated in both the nightly and
release publishing jobs of the workflow. Since the workflow already contains a job specifically for the purpose of
determining the characteristics of the build being performed and making that information available from single source
for use throughout the rest of the workflow, it makes sense to also move the S3 upload determination code to that job.
On every release tag, and manual trigger when the "Include builds on non-free runners" checkbox is checked, make the
Arduino IDE build for native Apple Silicon host in addition to the builds that are always generated by the "Arduino IDE"
GitHub Actions workflow.
Previously, the build workflow only produced a build for x86-64 (AKA "Intel") macOS hosts. Although it is possible to
use those builds on Apple Silicon machines via the Rosetta 2 translation software, the performance is significantly
inferior to a native build so we must also provide Apple Silicon native builds.
Previously the Apple Silicon builds were produced manually. The reason for using that inefficient and error-prone
approach instead of the automated continuous deployment system used for other builds was that GitHub did not provide the
necessary Apple Silicon runner machines and Arduino was not capable of setting up such self-hosted machines in a manner
that would make them feasible for the project maintainers to use. GitHub hosted Apple Silicon runner machines are now
available so we can add the target to the build workflow.
GitHub gives unlimited use of the basic runner machines for workflow runs in public repositories. However, the macOS ARM
architecture is only provided in runner machines which are classified as "larger runner". Use of these runners is
charged on a per-minute basis, without any of the free allowances GitHub provides for the normal runners. In order to
avoid unnecessary expenditures, native Apple Silicon builds must be generated only when there is compelling reason to do
so. Such a build is needed for every release, so the workflow is configured to always generate the builds when triggered
by a tag. In addition to releases, Apple Silicon tester builds for pull requests that might have special implications
for this target. For this reason, the workflow is configured to allow Apple Silicon builds to be triggered manually by a
repository maintainer.
The workflow uses a job matrix to run the build for each target on the appropriate runner machine in parallel, using the
universally applicable workflow code for all jobs. It uses another job matrix to generate individual workflow artifacts
for the tester builds of each target. Previously it was possible to always use the same matrix configurations for all
workflow runs. With the addition of the selectively run macOS ARM job, it is now necessary to generate these matrixes on
the fly.
The electron-updater package used by Arduino IDE's auto-update capability uses a data file (known as the "channel update
info file") to check for the availability of updates. A single "channel update info file" is used for the data of the
macOS x86 and ARM builds. Since a separate job is used to produce each of those builds, this means the "channel update
info file" produced by each of the macOS build jobs must be merged into a single file.
The "Arduino IDE" GitHub Actions workflow is used to generate several distinct types of builds:
- Tester builds of commits
- Nightly builds
- Release builds
Different actions must be performed depending on which type of build is being produced. The workflow contains code that
uses various criteria to determine the build type.
Previously that code was duplicated in multiple places:
- The packaging job
- The changelog generation job
- The nightly build publishing job
- The release publishing job
This duplication is avoided by moving the code to a dedicated job that makes the build type information available to all
subsequent jobs via outputs.
The "Arduino IDE" GitHub Actions workflow uses a job matrix to make the builds for each host target in parallel.
The same steps are used for each job, but some configuration adjustments must be made on a per-target basis. This is
done through various attributes in the matrix configuration.
Previously the `os` attribute was used for two distinct things:
- The machine identifier of the GitHub Actions runner machine of the job.
- The differentiator in the human-targeted job name.
The attribute name "os" (for "operating system") was misleading because runners are differentiated by more than only the
operating system on the machine.
The use of a machine identifier as a differentiator in the human-targeted job name was a bad idea because these
identifiers would only effectively communicate the nature of a build to humans who are quite knowledgeable about the
GitHub Actions workflow syntax.
The impact of these poor decisions has not been too severe previously due to there only being a single job for each
operating system. However, there is a need for multiple jobs per operating system in order to support multiple host
architectures (e.g., macOS x86 and ARM).
The solution is to:
- Use an appropriate name for the runner identifier attribute.
- Use a dedicated attribute for the human friendly job name differentiator.
- update status bar if board+port is restored,
- refresh the debug toolbar if board+port is restored,
- init `Include Library` if board+port is ready, and
- init library examples if board+port is ready
Closes#2237Closes#2239
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>
Store the board config data per sketch and not per IDE installation.
The (Theia) default `StorageService` implementation is workspace-scoped
when `@theia/workspace` is part of the final application.
Closes#2240
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>
- Update to `electron-builder@24.6.3`.
- Fix obsolete electron security section in the development docs.
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>
