Background
==========
Shared Library Dependencies
---------------------------
The Linux build of Arduino IDE has dynamic linkage against the libstdc++ and glibc shared libraries. This results in
it having a dependency on the version of the libraries that happens to be present in the environment it is built in.
Although newer versions of the shared libraries are compatible with executables linked against an older version, the
reverse is not true. This means that building Arduino IDE on a Linux machine with a recent distro version installed
causes the IDE to error on startup for users who have a distro with older versions of the dependencies.
For example, if Arduino IDE were built on a machine with version 3.4.33 of libstdc++, then attempting to run it on a
machine with an older version of libstdc++ would fail with an error like:
```
Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.33' not found (required by /home/foo/arduino-ide/resources/app/lib/backend/native/nsfw.node)
```
Likewise, if Arduino IDE were built on a machine with version 2.39 of glibc, then attempting to run it on a machine
with an older version of glibc would fail with an error like:
```
Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by /home/foo/arduino-ide/resources/app/node_modules/nsfw/build/Release/nsfw.node)
```
Build Machine Requirements
--------------------------
The IDE builds distributed by Arduino should be compatible with a reasonable range of Linux distribution versions. In
order to achieve this, the builds must be performed in a machine with an older version of the shared libraries. The
shared libraries are part of the Linux distro, and installing a different version is not feasible. So this imposes a
maximum limit on the build machine's distro version.
The distributed builds are generated via a GitHub Actions workflow. The most simple approach is to run the build in the
machine of the GitHub-hosted runners provided for each operating system. However, GitHub provides a limited range of
operating system versions in their runners, and removes the older versions as newer versions are added. This means that
building in the GitHub-hosted runner machine would not allow for the desired range of Linux distro version
compatibility. For this reason, the Linux build is performed in a Docker container that provides an older version of
Ubuntu.
The same situation of incompatibility with Linux distro versions that have a version of the shared library dependencies
older than the version present on the build machine occurs for several of the tools and frameworks used by the build
process (e.g., Node.js, Python). In this case, the tables are turned as we are now the user rather than the distributor
and so are at the mercy of the Linux distro version compatibility range provided by the distributor. So this imposes a
minimum limit on the build machine's distro version.
Although several of the dependencies used by the standard build system have dependencies on versions of glibc higher
than the version 2.27 present in Ubuntu 18.04, it was possible to use this distro version in the Linux build container
by using alternative distributions and/or versions of these dependencies.
Workflow Artifacts
------------------
The build workflow uses GitHub actions workflow artifacts to transfer the files generated by the build job to subsequent
jobs in the workflow. The "actions/upload-artifact" action is used for this purpose.
Problem
=======
GitHub is dropping support for the workflow artifacts produced by the version 3.x of the "actions/upload-artifact"
action that was previously used by the build job. So the action version used in the build workflow was updated to the
current version 4.x. This version of the action uses a newer version of the Node.js runtime (20). Unfortunately the the
Node.js 20 runtime used by the action has a dependency on glibc version 2.28, which causes the Linux build job to fail
after the update of the "actions/upload-artifact" action:
```
Run actions/upload-artifact@v4
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
```
Unlike the other dependencies of the build process, it is no longer possible to work around this incompatibility by
continuing to use the older compatible version of the "actions/upload-artifact" action. It is also impossible to replace
the incompatible Node.js 20.x distribution used by the action, since it comes from the read-only file system of the
runner image. Likewise, it is not possible to configure or force the action to use a Node.js installation at a different
path on the runner machine.
Resolution
==========
Compatibility with the new version of the "actions/upload-artifact" action is attained by updating the version of Linux
in the build container to 18.10, which is the oldest version that has glibc 2.28. The presence of a newer glibc version
in the container also makes it compatible with several other dependencies of the build process, meaning the code in the
Dockerfile and workflow for working around the incompatibilities of Ubuntu 18.04 can be removed.
Consequences
============
Unfortunately this means the loss of compatibility of the Linux Arduino IDE builds with distros that use glibc 2.27
(e.g., Ubuntu 18.04). User of those distros will now find that Arduino IDE fails to start with an error like:
```
Error: node-loader:
Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /home/foo/arduino-ide/resources/app/lib/backend/native/pty.node)
at 85467 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:2766)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
at 23571 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:3374073)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
at 55444 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:3369761)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
at 24290 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:1780542)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
at 43416 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:1770138)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
```
The build workflow produces binaries for a range of target hosts. This is done by using a job matrix in the GitHub
Actions workflow that produces each build in a parallel job. GitHub Actions workflow artifacts are used to transfer the
generated files between sequential jobs in the workflow. The "actions/upload-artifact" action is used for this purpose.
Previously, a single artifact was used for this purpose, with each of the parallel jobs uploading its own generated
files to that artifact. However, support for uploading multiple times to a single artifact was dropped in version 4.0.0
of the "actions/upload-artifact" action. So it is now necessary to use a dedicated artifact for each of the builds.
These can be downloaded in aggregate by using the artifact name globbing and merging features which were introduced in
version 4.1.0 of the "actions/download-artifact" action.
The "Sync Labels" GitHub Actions workflow is configured to allow the use of multiple shared label configuration files.
This is done by using a job matrix in the GitHub Actions workflow to download each of the files from the source
repository in a parallel GitHub Actions workflow job. A GitHub Actions workflow artifact was used to transfer the
generated files between sequential jobs in the workflow. The "actions/upload-artifact" and "actions/download-artifact"
actions are used for this purpose.
Previously, a single artifact was used for the transfer of all the shared label configuration files, with each of the
parallel jobs uploading its own generated files to that artifact. However, support for uploading multiple times to a
single artifact was dropped in version 4.0.0 of the "actions/upload-artifact" action. So it is now necessary to use a
dedicated artifact for each of the builds. These can be downloaded in aggregate by using the artifact name globbing and
merging features which were introduced in version 4.1.0 of the "actions/download-artifact" action.
- Use the CLI release proto.zip to get proto files for production versions of CLI
- Extract the proto files from repo if CLI version is declared as `commitsh` or version is 1.1.0
See https://github.com/arduino/arduino-cli/pull/2761
Adds a new preference to control whether the
verify command should automatically run before the
upload. If the `arduino.upload.autoVerify` setting
value is `false`, IDE does not recompile the
sketch code before uploading it to the board.
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* fix: use `Status` enum for status code in `ServiceError` type guards
This change resolves the issue where the intersection of `ServiceError` error codes of type `number` resulted in the `never` type due to conflict between number and `State` enum if `StatusObject`
* feat: add `isInvalidArgument` type guard to `ServiceError`
* fix: retry compilation if grpc client needs to be reinitialized
See https://github.com/arduino/arduino-ide/issues/2547
It is considered good practice to use JavaScript's type-safe strict equality operator === instead of the equality
operator ==. Compliance with this practice is enforced by the project's ESLint configuration, via the "eqeqeq" rule.
The script used to generate the changelog for Arduino IDE's auto-update dialog contained an inappropriate usage of the
equality operator. This caused linting runs to fail:
arduino-ide-extension/scripts/compose-changelog.js
37:19 error Expected '===' and instead saw '==' eqeqeq
The `lint` script of the "arduino-ide-extension" package is intended to use the ESLint linter tool to check for problems
in all the package's JavaScript and TypeScript code files. It is used by the continuous integration system to validate
contributions.
Previously, the command invoked `eslint` without any arguments. With the 8.x version of ESLint used by the project, it
is necessary to provide a path argument in order to cause it to lint the contents of files. Because that argument was
not provided, the script didn't do anything at all and so would return a 0 exit status even if the code had linting rule
violations.
This is fixed by adding a `.` path argument to the command invoked by the script. This will cause ESLint to recurse
through the `arduino-ide-extension` folder and lint the code in all relevant files.
The "build" workflow builds the application for all supported targets, generates workflow artifacts from which the
builds can be downloaded by users and beta testers, and publishes nightly and production releases.
As if that wasn't enough, the workflow was also configured to check the sync of the Yarn lockfile.
This monolithic approach is harmful for multiple reasons:
* Makes it difficult to interpret a failed workflow run
* Makes the build workflow more difficult to maintain
* Increases the turnaround time for contributors and maintainers to get feedback from the CI system
The sync check operation is hereby moved to a dedicated workflow, consistent with standard practices for Arduino Tooling
projects.
The "build" workflow builds the application for all supported targets, generates workflow artifacts from which the
builds can be downloaded by users and beta testers, and publishes nightly and production releases.
As if that wasn't enough, the workflow was also configured to perform the unrelated operation of running the project's
test suites.
This monolithic approach is harmful for multiple reasons:
* Makes it difficult to interpret a failed workflow run
* Unnecessarily adds a significant amount of extra content to the already extensive logs produced by the build process
* Makes the build workflow more difficult to maintain
* Increases the length of a build workflow run
* Increases the impact of a spurious failure
* Increases the turnaround time for contributors and maintainers to get feedback from the CI system
The test run operation is hereby moved to a dedicated workflow, consistent with standard practices for Arduino Tooling
projects.
The "build" workflow builds the application for all supported targets, generates workflow artifacts from which the
builds can be downloaded by users and beta testers, and publishes nightly and production releases.
As if that wasn't enough, the workflow was also configured to perform the unrelated operation of linting the project's
TypeScript and JavaScript code.
This monolithic approach is harmful for multiple reasons:
* Makes it difficult to interpret a failed workflow run
* Unnecessarily adds a significant amount of extra content to the already extensive logs produced by the build process
* Makes the build workflow more difficult to maintain
* Increases the length of a build workflow run
* Increases the impact of a spurious failure
* Increases the turnaround time for contributors and maintainers to get feedback from the CI system
The linting operation is hereby moved to a dedicated workflow, consistent with standard practices for Arduino Tooling
projects.
The "build" workflow signs the macOS and Windows builds of the application. The signing process relies on access to GitHub Actions
secrets. For this reason, the workflow is configured to only sign the builds when it has access to GitHub Actions
secrets to avoid spurious failures of the workflow that would otherwise be caused by signing failure.
A flexible general purpose system for determining whether to attempt signing of a build was established years ago. However, a redundant system was added specific to the Windows build instead of using the existing system.
The redundant system is hereby removed. This makes the workflow easier to understand and maintain.
The "build" workflow signs the Windows builds of the application. The signing process relies on access to GitHub Actions
secrets. For this reason, the workflow is configured to only sign the builds when it has access to GitHub Actions
secrets to avoid spurious failures of the workflow that would otherwise be caused by signing failure.
Previously the signing was determined based on the value of the `github.event.pull_request.head.repo.fork` context item.
That was effective for the use case of the workflow being triggered by a pull request from a fork (for security reasons,
GitHub Actions does not give access to secrets under these conditions).
However, there is another context under which the workflow might run without access to the signing secrets, for which
the use of context item is not appropriate. It is important to support the use of the workflow in forks of the
repository. In addition to the possible value to hard forked projects, this is essential to allow conscientious
contributors to test contributions to the build and release system in their own fork prior to submitting a pull request.
The previous configuration would cause a workflow run performed by a contributor in a fork to attempt to sign the
Windows build. Unless the contributor had set up the ridiculously complex infrastructure required to perform the signing
for the Windows build, which is utterly infeasible, this would cause the workflow to fail spuriously.
The appropriate approach, which has been the established convention in the rest of the workflow code, is to use the
secret itself when determining whether to attempt the signing process. If the secret is not defined (resulting in it
having an empty string value), then the signing should be skipped. If it is defined, then the signing should be
performed.
The "build" workflow builds the application for a range of target hosts. This is done by using a job matrix. A separate
parallel job runs for each target. The target-specific configuration data is defined in the job matrix array.
This configuration data includes the information related to the code signing certificates. Inexplicably, during the work
to add support for signing the Windows builds with an "eToken" hardware authentication device, this data was not used
for the Windows code signing configuration. Instead the certificate data was redundantly hardcoded into the workflow
code.
The Windows code signing certificate configuration is hereby changed to use the established flexible job configuration
data system. This makes the workflow easier to understand and maintain.
The Windows builds of the application are cryptographically signed. The signing requires an "eToken" hardware
authentication device be connected to the machine performing the signing. This means that it is necessary to use a
self-hosted GitHub Actions runner for the Windows job of the build workflow rather than the runners hosted by GitHub.
There are some unique characteristics of the self-hosted runner which the workflow code must accommodate. One of these
is that, rather than installing dependencies of the build process during the workflow run as is done for the
GitHub-hosted runners, the dependencies are preinstalled in the self-hosted runner machine. So the dependency
installation steps must be configured so that they will be skipped when the job is running on the self-hosted runner.
This is done by adding a conditional to the steps. Previously the conditional was based on the value of the `runner.os`
context item. This is not an appropriate indicator of the job running on the self-hosted runner because `runner.os` will
have the same value if the job was running on a GitHub-hosted Windows runner. That might seem like only a hypothetical
problem since the workflow does not use a GitHub-hosted Windows runner. However, it is important to support the use of
the workflow in forks of the repository. In addition to the possible value to hard forked projects, this is essential to
allow conscientious contributors to test contributions to the build and release system in their own fork prior to
submitting a pull request.
The conditionals are changed to use the more appropriate indicator of the specific name of the self-hosted Windows
runner (via the `runner.name` context item).
The Windows builds of the application are cryptographically signed. The signing requires an "eToken" hardware
authentication device be connected to the machine performing the signing. This means that it is necessary to use a
self-hosted GitHub Actions runner for the Windows job of the build workflow rather than the runners hosted by GitHub.
There are some unique characteristics of the self-hosted runner which the workflow code must accommodate. The default
working directory of the self-hosted runner is not suitable to perform the build under because the the resulting folder
structure produced paths that exceeded the ridiculously small maximum path length of Windows. So the workflow must be
configured to use a custom working directory with a short path (`C:\a`).
This custom working directory must be used only for the job running on the self-hosted Windows runner so the working
directory of the relevant workflow steps are configured using a ternary expression. Previously, this expression had
multiple conditions:
* the value of the `runner.os` context item
* the definition of a custom working directory value in the job matrix
The second condition is entirely sufficient. The use of the first condition only added unnecessary complexity to the
workflow code, and imposed a pointless limitation of only allowing the use of the custom working directory system on
Windows runners.
Removing the unnecessary condition makes the workflow easier to understand and maintain, and makes it possible to
configure any job to use a custom working directory if necessary.
The build workflow produces builds for a range of target host architectures, including macOS Apple Silicon. This is done
by running a native build in a machine of the target architecture.
At the time the support for producing Apple Silicon builds was added to the workflow, use of GitHub-hosted Apple Silicon
runner machines was charged by the minute (while use of the other runners is free). In order to avoid excessive
expenses, the workflow was configured so that the Apple Silicon builds were only produced when absolutely necessary.
This was done by defining two sets of job matrix arrays, one for jobs using free runners, and the other for jobs using
paid runners. Due to the limitations of the GitHub Actions framework, it was necessary to use workflow environment
variables for this purpose.
Since that time, GitHub made free GitHub-hosted Apple Silicon runners available. When the workflow was adjusted to use
that runner, the configuration for the Apple Silicon build job was moved to the free runner job matrix array. The system
for supporting selective use of paid GitHub-hosted runners to produce builds was not removed at that time. This is
reasonable since it is possible the need will arise for using paid runners at some point in the future (e.g., only
legacy older versions of free macOS "Intel" runners are now provided and it is likely that even these will eventually be
phased out forcing us to use the paid runner to produce builds for that target). However, the environment variable for
the paid runner job matrix array data was removed. The absence of that variable made it very difficult to understand the
workflow code for the system.
For this reason, the environment variable is replaced, but empty of data. A comment is added to explain the reason for
this.
Document prerequisites of the Arduino CLI, LS, etc. tools when built
from a Git commitish.
---------
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
Co-authored-by: per1234 <accounts@perglass.com>
On every startup, Arduino IDE checks for new versions of the IDE. If a newer version is available, a notification/dialog
is shown offering an update.
"Newer" is determined by comparing the version of the user's IDE to the latest available version on the update channel.
This comparison is done according to the Semantic Versioning Specification ("SemVer").
In order to facilitate beta testing, builds are generated of the Arduino IDE at the current stage in development. These
builds are given an identifying version of the following form:
- <version>-snapshot-<short hash> - builds generated for every push and pull request that modifies relevant files
- <version>-nightly-<YYYYMMDD> - daily builds of the tip of the default branch
In order to cause these builds to be correctly considered "newer" than the release version, the version metadata must be
bumped immediately following each release.
This will also serve as the metadata bump for the next release in the event that release is a minor release. In case it
is instead a minor or major release, the version metadata will need to be updated once more before the release tag is
created.
Status object thrown by grpc commands contains metadata that needs to be serialized in order to map it to custom errors generated through proto files https://github.com/grpc/grpc-web/issues/399
* fix: use `@pingghost/protoc` to compile proto files
The npm package previously used (`protoc`) is still lacking apple arm32 support, see https://github.com/YePpHa/node-protoc/pull/10
* feat: use Arduino CLI 1.0.4
* fix: allow use of node16 in github actions
* chore: update `arduino-language-server` version for cli-1.0.0
* fix: deprecated platform order test
Arduino deprecated platforms should have more priority then other deprecated ones
