After merging the last PR, I was rather eager to record a live demo from a mobil…e device, as well as publicly announce my progress with the project and its ultimate motivation. Yet, I was almost foiled again by numerous corner cases: e.g. what would work fine from my local workstation would not work on mobile, and what did work on mobile using remote Codespaces would not work from my local workstation. While I managed to resolve much of the friction and inconsistencies with PWA installation and runtime UX, I'm guessing others who attempt to replicate this work may encounter similar gotchas. So, I'll document my findings here in detail, in hopes that it may help others in the future.
## PWA Installation
For any PWA to be properly installable, the WC3 specification mandates some installation criteria; a manifest obviously being one of them, both also a few others not as immediately apparent. For a complete list of requirements, you can refer to the following documentation:
- WC3: Web App Manifest
- https://www.w3.org/TR/appmanifest/
- Web.dev: PWA Installation criteria
- https://web.dev/learn/pwa/installation/#installation-criteria
While in theory the satisfaction of every criteria is required, in practice this is not yet the case, and perhaps thankfully so. For example, Chrome does not yet require a registered service worker, nor always require https connections, either of which could end up adding a bit more complexity to our existing setup. However, there are few we'll need to truly satisfy, lest we leave users with only a partial PWA experience.
When "installing" web apps from the browser, users may be presented with different options, such as "Add to Home screen" on mobile or "Create Shortcut..." on desktop. These options are similar to adding a hyperlink or bookmark URL to the operating system's desktop. However, when installing a proper PWA, users are instead presented with an "Install App" option or even a toast prompt recommending them to do so. While, for our current apps, both options can ultimately provide the same experience for desktop users, enabling the "Open as Window" option when creating shortcuts will not have the same effect for mobile users, particularly on Android and more affordable devices. Only proper PWAs can open separate windows from the browser on these devices.
This subtle distinction initially led me to believe that PWA installation was working on desktop and would be ready for testing on mobile as well. I also believed that PWA shortcuts were still too experimental to be available on stable browser releases because when long-pressing the web app icons on the home screen, the additional shortcuts specified in the PWA manifest did not appear, as they do for native mobile apps. This difference was precisely due to the distinction between "Add to Home screen" and "Install App," as indicated by the badging of the web browser icon on top of the web app's bookmark icon.
### Icons
The first criterion we can address is the inclusion of icons of suitable size and format. While most of the existing web app directories ship with a favicon.ico file for the browser to display on the tab bar, the nav2 repo does not include one, and the .ico file format is not supported by the PWA manifest. Although the nav2 logo is conveniently available in the repo as a .png file, it is not square in resolution. Therefore, for demo purposes, I quickly created and formatted the necessary icons for each app. I used an online tool called Favicon Generator to manage this process for most platforms. You can find it here:
- Favicon Generator. For real.
- https://realfavicongenerator.net/
While we could simply commit the generated icons directly, it is generally not advisable to merge binary files into a Git repo. A better practice would be to use a Git LFS (Large File Storage) service, such as the one provided natively with GitHub. However, the free tier of GitHub LFS only allows for 1GB of storage and bandwidth per organization, including all derivative forks. Given the large size and popularity of the ros-planning organization, relying on not exceeding this quota would be risky.
- Git LFS
- https://git-lfs.com/
- GitHub LFS: About storage and bandwidth usage
- https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-storage-and-bandwidth-usage
A simpler solution is to host any media files externally and fetch them during the Docker image build process for the Dev Container. In this case, we can attach the icon pack archive to GitHub and use the anonymized URL to download the archive during the build process. Of course, we'll also check the checksum before extraction to ensure that the archive link remains valid and that we don't download anything unexpected.
- GitHub Docs: Attaching files
- https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/attaching-files
Note that it's beneficial to keep such online build steps separate from the container creation process. Otherwise, it can lead to dependencies on online resources even when recreating the Dev Container. For example, it would be preferable if building and developing Nav2 using Dev Containers remained possible when offline or with limited internet connectivity, once the Docker image has already been built and distributed to students.
While it may seem like a lot of effort for just a few hundred kilobytes of icons, we can easily replicate this media management pattern for larger assets that we prefer not to commit, such as example bag files or datasets for students to practice with. However, for anything larger than a few tens of megabytes, using volumes or bind mounts would be a better option. For now, we can address the issue of not knowing which shell or command line tools are preinstalled on the students' host machines by simply baking the icons into the image when building with Docker.
Regarding file size, we can also speed up initial page loads for web apps by compressing these icons. Even with lossless format settings, we can achieve compression rates of over ~80% compared to stock PNG exports. This allows us to use two moderately sized icons for each app: a maskable version for mobile and an enlarged transparent version for any other use cases. Although maskable icons are optional, they provide a much nicer appearance on mobile. The following online documentation and tools provide more examples:
- Adaptive icon support in PWAs with maskable icons
- https://web.dev/maskable-icon/
- Maskable.app
- https://maskable.app/
- ShortPixel: Online Image Compressor
- https://shortpixel.com/online-image-compression
### Manifest
The next criterion to address is the manifest file. Although these assets were added with the previous PR, I still couldn't properly install any of the PWAs on mobile, even after resolving the icon criterion. This was due to a number of compounding issues. Firstly, PWAs can only be installed from origins that are served over HTTPS. A notable exception is when developing locally, where the browser allows for the installation of PWAs from `localhost` or `127.0.0.1`. When testing locally using separate hosts and clients, there are several ways to resolve this:
#### Local Port Forwarding
If the device supports any form of local port forwarding, such as SSH, we can simply forward a local port on the device to the port of the Caddy web server running in the Dev Container. This can be done from a local workstation. While this may be more involved for young students and would need to be repeated for each new device session, it is possible even on Android or Chromebook devices. One only needs to install an app or enable Linux-like features, thus allows mobile browsers to install locally hosted PWAs using only HTTP. Although I'm not sure what the equivalent workaround would be for the iOS platform.
- Termux Wiki
- https://wiki.termux.com/wiki/Remote_Access
- Android App: JuiceSSH
- https://juicessh.com/
- How to use the handy SSH management tool in Chrome OS
- https://www.zdnet.com/article/how-to-use-the-handy-ssh-management-tool-in-chrome-os/
#### SSL certificates
A more scalable approach would be to properly serve the PWAs using HTTPS. For institutions that manage their own IT infrastructure and device administration, self-signed root certificates could be used. If you own a URL domain, you could also use a separate Caddy instance to host your own HTTPS reverse proxy in front of any Dev Containers. Caddy makes this process rather easy with its Automatic HTTPS feature.
- Caddy Docs: Automatic HTTPS
- https://caddyserver.com/docs/automatic-https
This is the approach I took when debugging the HTTPS issues I mentioned earlier. Having a working HTTPS session external to Codespaces was valuable in isolating corner cases between using just the HTTPS protocol and using HTTPS with GitHub's authentication proxy. In fact, here's the gist of my entire Caddyfile used for that purpose:
```Caddyfile
https://example.com {
reverse_proxy * "localhost:8080"
}
```
### Cross Origin
While the HTTPS criterion and the subsequent exception for the localhost origin explained why I could install the PWAs hosted from any local Dev Container using my workstation's browser, but not from mobile, it didn't explain why I couldn't install the PWAs from Codespaces.
As explained in the previous PR, ports forwarded to the web from Codespaces are served by default using HTTPS. Although this default setting is labeled as HTTP, it means that traffic forwarded to Dev Container ports will be HTTP, with GitHub's HTTPS proxy handling TLS connections externally using its own valid Origin certificates. In theory, this should have been sufficient for PWA installation. However, in practice, when inspecting the web app page under the "Application" tab using Chrome DevTools, no manifest was detected.
Despite the manifest link being included in the HTML response, and the URI to the manifest responding with the expected JSON content, it was only when inspecting the "Network" tab and comparing the request headers between fetching the manifest file from Codespaces by loading the web page for the PWA and loading the URI directly to the JSON file that I noticed the absence of authentication cookies in the request headers of the former.
- Chrome DevTools Docs: Debug Progressive Web Apps
- https://developer.chrome.com/docs/devtools/progressive-web-apps/
The absence of authentication cookies in the request header resulted in cryptic redirects when attempting to fetch the manifest from the same origin as the web app. This even led to Cross Origin policy violations as GitHub's authentication proxy redirected to an error page. It wasn't until I tried hosting my own HTTPS proxy that this became apparent. Additionally, it revealed another proxy header to check when determining the session protocol from the client's perspective: `X-Forwarded-Proto`.
Upon revisiting the MDN docs, I found the following caveat specifically regarding deploying the manifest, which added to our second compounding issue. I can only guess that this was chosen to provide simpler support for deploying PWAs using Content Delivery Networks (CDNs), allowing installation from distributed caches without exposing session credentials.
> If the manifest requires credentials to fetch, the crossorigin attribute must be set to use-credentials, even if the manifest file is in the same origin as the current page.
```html
<link rel="manifest" href="/app.webmanifest" crossorigin="use-credentials" />
```
- MDN Docs: Deploying a manifest
- https://developer.mozilla.org/en-US/docs/Web/Manifest#deploying_a_manifest
With this simple change, we can finally authenticate when requesting manifest files, providing us with our long sought after `Install App` menu option in the browser.
### PWA Scope
Now that we have resolved the last PWA installation criterion, we can install any of our self-hosted PWAs. However, we still face a challenge: we can't install all the self-hosted PWAs simultaneously. When attempting to install a second PWA, browsers suggest reopening the page in the first installed PWA. This is partly due to how browsers uniquely identify PWAs, particularly when no `id` or `scope` parameters are specified. In such cases, browsers fallback to using the `start_url` parameter. While the `start_url` parameters were unique for each PWA, the URI directory (which the browser uses to infer scope) was not, as it was always root `/`. Thus simple omission of a trailing slash was yet again the "root" of another issue. See the example and references below for more details:
```diff
{
"name": "Foxglove: {{placeholder "http.vars.ReqHost"}}",
...
+ "id": "/foxglove/",
- "start_url": "/foxglove",
+ "start_url": "/foxglove/",
...
```
- Uniquely identifying PWAs with the web app manifest id property
- https://developer.chrome.com/blog/pwa-manifest-id/
By including the `id` parameter and the trailing slash to denote each PWA's unique scope, we can now install all PWAs simultaneously. However, there is another caveat: we can only install them in a particular order. This is because of how I initially structured the PWAs using overlapping/nested paths, which turned out to be a mistake. While I had already segmented each PWA into separate non-overlapping paths (e.g., `/foxglove/` and `/gzweb/`), this was not the case for the landing page, or what I'll now refer to as the Nav2 PWA launcher, which was hosted at the root path `/`.
The approach of using overlapping paths (or scopes) for PWAs served from a shared common origin can be quite tricky and is best avoided altogether. However, hosting each PWA over separate ports, and thus separate origins, also comes with its own complications, including the added complexity of managing the Caddyfile. Therefore, we will stick with using a single origin but ensure that all PWAs, including the Nav2 App launcher, have strictly non-overlapping paths. So, we will need to reorganize the orchestration to achieve this. For more details on why, refer to the references below:
- Progressive Web Apps in multi-origin sites
- https://web.dev/multi-origin-pwas/
- Building multiple Progressive Web Apps on the same domain
- https://web.dev/building-multiple-pwas-on-the-same-domain/
## Organization and Orchestration
To avoid the complications of overlapping scopes, we need to reorganize the orchestration of our PWAs. This involves moving the Nav2 App launcher to a separate path instead of the root URI. Additionally, after careful thought and experimentation, it seems that using a single file server with symbolic links to a single root directory was indeed the simplest approach for hosting multiple PWAs from a single origin.
### Caddyfile
To achieve this, while still maintaining the convenience of the Nav2 PWA launcher, we can move the launcher to the `/nav2/` path and add a permanent redirect from `/` to `/nav2/` when the user first navigates to the root URI of the host port origin. This prevents the launcher's scope from interfering with the scopes of nested PWAs, while ensuring the launcher remains easily discoverable.
While I did waffle around between various approaches such as using symbolic links, multiple environment variables, and `file_server` directives. Eventually, I settled on a hybrid approach that maintains runtime flexibility while simplifying the configuration. We can use an environment variable to configure the root for the file server and create symlinks from there to link to the respective PWA directories, whether within the project workspace or elsewhere in the container/host. This allows us to further simplify our Caddyfile into a more generic template.
Furthermore, this approach enables us to leverage the convenient `browse` feature supported by Caddy's `file_server` directive. We can now have a simple file browser for the entire site without indexes, which is particularly useful for debugging. Previously, the directory hierarchy posed limitations on the functionality of the `browse` feature since multiple `file_server` directives were used. By modifying the matcher of the permanent redirect for the Nav2 launcher, we can create an exception using a URL query parameter to browse the entire root directory and view the complete site index.
```Caddyfile
# Matcher for requests without browse query
@no_browse {
path /
not query browse=true
}
# Redirect to nav2 web app by default
redir @no_browse /nav2/
```
- Caddy Docs: file_server
- https://caddyserver.com/docs/caddyfile/directives/file_server
### Miscellaneous
Now that the installation process is running smoothly, we can further enhance the manifest file. Adding a theme color and background colors that match the maskable icon background will give the loading splash screen a cool appearance. Additionally, we can include shortcuts to other PWAs, such as setting variable refresh rates for the system monitor. Furthermore, we will upgrade all the other apps, except for the Nav2 launcher, to display in fullscreen mode instead of the standalone view. This will optimize the use of screen real estate for visualizations.
I also attempted to add a file browser shortcut to the Nav2 launcher PWA. However, it's important to note that PWAs can only navigate to URIs within their declared scopes. While we could change the scope of the Nav2 launcher to include the entire root directory and render the file browser within the same standalone window, this would negatively impact the user experience during PWA installations, and would reintroduce the confusion we were trying to avoid initially, making it not worth the effort. Besides, deferring to a browser UI for such paths is not significantly inconvenient.
