Rewrite, improve, update and correct the documentation

[Korne127]

This is still a work in progress. However, I now got most of the major changes done, and since I might still take a while for the rest and am a bit afraid that someone else might want to start doing the same, I decided to open the current state as a draft pull request.

---

When I started looking into how to deploy AppImages a few weeks ago, I had big troubles understanding the documentation:

• There was very much duplicated information which made it hard to find a specific detail that was only mentioned in one of the duplicated parts and generally to read through the documentation.

• There was very much outdated information, from commonly recommending the deprecated linuxdeployqt tool, to the deprecated AppRun.c file, not even mentioning the new go-appimagetool, many invalid and outdated links, old Linux versions (one even being eleven years (!) old), etc.

• The same terms were sometimes used for very different concepts, e.g. "manual packaging" for

  1. manually creating the entire AppDir that's then converted into an AppImage by appimagetool,
  2. manually creating the AppDir structure to use with higher level tools that include the shared libraries and
  3. giving the path of executables, desktop and icon files to linuxdeploy via command line arguments

  which obviously led to much confusion.

• There was much conflicting information, mostly due to duplicated information, which has been outdated in some parts and updated in others, or the same terms used for different concepts.

• There was missing information that was expected in other places, e.g. which resolutions are valid for icon files.

• There was information scattered across places or in the wrong places, e.g. several pages contained some details about desktop files, but no page in the packaging guide clearly explained everything related to it and how to create them.

• There were weird structure decisions, e.g. placing a software overview, mostly about low-level software that the average user should not use, in the introduction section instead of the reference section.

• There was no single detailed comparison about the differences between the individual tools and approaches to create an AppImage, resulting in a lot of confusion which one(s) to use and how to create AppImages.

• Some things were just explained badly, e.g. the pages about using make to create an AppDir lacked the most important information on how to do so, and the linuxdeploy page made it very hard to understand how to use it without make.

I think much of this is not that noticeable when you know your way around AppImages well enough. Sure, you might notice many things aren't up to date and that some things are duplicated, but I think you can generally read through it, understand what's meant, fill in the missing information and don't really realize how hard it is to understand all of that if you don't have that prior experience and knowledge.

I also think that much of this comes from the fact that the AppImage ecosystem has evolved a lot in the past years. There are many things that have considerably improved, e.g. going from having to package the entire AppDir by yourself and use appimagetool directly, over tools like linuxdeployqt that package all shared libraries from an existing AppDir structure, to linuxdeploy, which can get all necessary data with command line arguments and requires no pre-existing structure.
I also noticed that some old parts were expecting the reader to repackage existing packages, probably out of a time where AppImages haven't been commonly used, while other parts were mainly directed to the application authors.
I can imagine that the focus was on developing better solutions and improving the AppImage packager and user experience. And it's understandable that, with many parts duplicated, you don't always find the time to really go through the documentation and update everything correctly when something changes, slowly leading to a worse documentation state.

All of this is to say, I hope that you don't take this as an insult. I do appreciate the work you put into AppImages and the whole ecosystem and I understand that the focus has been on other things, but I do think that the documentation is in a bad state.

And this also led to frustration by other users. See for example https://discourse.appimage.org/t/does-appimage-include-all-shared-library-dependencies-by-default/509/6, where someone is frustrated with the documentation as they don't find a guide on how to automatically package shared libraries, or ruffle-rs/ruffle#12401, where someone didn't even know about the other available tools that help with this, which is the core thing the documentation should explain in my opinion.

---

Therefore I started to rewrite, improve, update and correct the documentation.
I started with the linuxdeploy page, which I had personally had a lot of struggle with to understand previously. From there on, I worked my way through the documentation (mostly the packaging guide), and went on to improve the structure, add a comparison of creation tools & approaches, deliver better explanations, update the outdated information and add additional information.

Here is an overview of my changes:

New sections that contain collected information

• One unified section for all AppImage creation tools has been added.
  • Its main page contains one central table which lists the advantages, disadvantages and differences between the different methods.
  • It contains one page for each AppImage creation tool (some of which have been collected from many places throughout the documentation), explaining how to use it.
    • This also means that a new page for go-appimagetool has been added (it had previously not been mentioned).
    • Much additional information and better explanations have also been added to most of these pages.
    • The linuxdeploy page has been rewritten, improved and updated. It’s now much easier to understand how to use it.
  • The section also contains a better overview on what the AppImage creation tools actually do, which helps to understand why they should be used.
• The information about the different packaging approaches (packaging as application author and repackaging existing binaries) has been collected in one section and rewritten, with additional information and a better explanation of the differences between the approaches.
• One unified section on how to create the AppDir structure has been added.
  • Its main page provides an overview of the different ways to do it, the reason why one would want to do it and the additional possibility of manually packaging everything.
  • This also means that the concepts of manually creating the AppDir structure (to give it to a tool) and manually packaging everything (without a tool) have been separated and the difference is well-explained.
• A new page about desktop entry files and icon files has been created. Information from several files has been merged together with much additional information.

Other sections that have been rewritten

• The packaging automation section (formerly called hosted services) has been rewritten and updated.
  • Its main page now contains a new introduction about the advantages of packaging automation and why it should be used.
  • A new tutorial on GitHub Actions has been added to it.
  • The existing (outdated) parts have been updated.
• The sections about which dependencies are bundled and which Linux distribution version the binaries should be compiled on in the concept page have been partially rewritten.
  • Misleading headlines have been changed and a better explanation of why AppImages should usually be compiled on older systems, and which exceptions and other options exist, has been added.
• The packaging guide introduction has been rewritten to give a better introduction of what the guide explains.

Updated & improved sections

• The AppDir specification has been updated.
  • Information about the AppRun file has been updated and added.
  • More information about the desktop entry and icon files has been added.

Split up sections

• The software overview section has been split up. Most of it has been moved into the reference section, but some parts moved into other sections in which the respective software had already been mentioned. A better introduction has been added to it as well.

Minor improvements

• The table formatting has been improved. Tables now have automatic line breaks (instead of scroll bars) and are easier to edit. (This is minor for the existing documentation, but the overview table wouldn’t have been possible without it).
• Many other sections and pages that haven’t been explicitly mentioned have gotten some improvements and additional information.

For more details, each commit contains an in-depth message of all changes I have made within it.

Also, this pull request documents the features introduced in linuxdeploy/linuxdeploy#288, so it depends on that being merged.

---

As I said in the beginning, this is currently a work in progress. I still have a lot of todos and there are quite some sections I haven't really gone through yet at all.
But I still thought that now is a good time to create this draft PR, as most of the major restructuring is done and it's easy to see my desired end result. I'll probably still take a while to finish this (especially as I basically worked the entire last week from waking up until going to bed on this, and will continue slower as it's now known that I'm working on it).

[Samueru-sama]

I support this, recently there was this discussion asking about the static and deploy everything tools: https://github.com/orgs/AppImage/discussions/1343

Btw a small nitpick, the part that says:

can also include these expected core libraries. This considerably increases the AppImage size (by at least 30MB)

Can be changed to say instead by at least 10 MiB and I've found cases where it is even less than that.

[Korne127]

@Samueru-sama I'm sorry for the late answer, I was really busy the last days.
Thanks for the notice! I'll definitely update the documentation about the new / different runtimes, especially as currently, all information about the different runtimes is not well documented and rather hard to understand. As I said, this is still a work in progress, and while I took a pause, I'll continue on this soon. Thanks for bringing it to my attention. :)

On the other part, I took that information from the appimage-builder documentation. But thanks for the feedback; I'll change the part to say that it's usually a big increase, but can also be smaller, depending on which libraries are called.

[Korne127]

@probonopd Hey, did you see this PR yet? I noticed that you were also working on the documentation in #103. While luckily, this doesn't interfere with my PR as it's a new section, I would have expected some kind of reaction or response to this.

[probonopd]

Hi @Korne127, thanks for this massive undertaking.

Hi @TheAssassin is there any way to get this documentation rendered out for easier review (without overwriting the current one, s that I can open them side by side)?

[probonopd]

Hi @Korne127

This considerably increases the AppImage size (by at least 30MB)

Where did you get this number from? If I remember correctly, the results i got were around 1/3rd that number. How did you test?

[Korne127]

@probonopd One way to look at the result would be to just clone my version and execute it:

git clone https://github.com/Korne127/docs.appimage.org.git
cd docs.appimage.org
mkdir venv
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
make html

And then the files are build in build/html, you just need to double-click them to open in a browser. I also discovered there is a make.sh script doing the (I think) five last steps for you.

[Korne127]

Where did you get this number from?

I took them from the appimage-builder documentation. If the numbers are wrong and your tests consistently produced different results, I'll adapt it.

(But keep in mind this is still a WIP and I haven't gone over some pages at all yet. I also plan to go through the project READMEs and updated them accordingly, e.g. if they link to a now unexacting link in the documentation.)

[probonopd]

Note to self,

sudo apt-get update
sudo apt-get install -y python3-pip python3.10-venv git
git clone https://github.com/Korne127/docs.appimage.org.git
cd docs.appimage.org
mkdir venv
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
make html
open html/index.html

works nicely for me. Maybe we should even document it ;)

[Korne127]

I finally continued on this, and went through the whole introduction and user guide pages and improved, rewrote, updated and corrected them all. I've spent most of my free time in the past week in that process. 😅 (I also merged two pages that basically provided the same kind of information in the same structure with big parts duplicate.) To view the exact changes and things I did, see the commit messages.
I think I'm about done 80%? There is still quite something on my to do list, I've gone through most pages now. Hopefully I'll finish this sometimes soon, because I think this is desperately needed.

Also, I noticed something important: The FUSE page in the AppImage wiki (https://github.com/AppImage/AppImageKit/wiki/FUSE) has generally the same structure behind its documentation page, but has been much more up-to-date (although it also missed some information).
Nevertheless, I think that there are many places in which things are documented: The AppImage wiki, the official documentation, the README files of many projects, other wikis that exist for these projects, etc. And because of that, information just gets updated in one place, which leads to more and more wrong or outdated information piling up.
I think we should place all up-to-date information in one place again (that's essentially what this PR does), and then remove the other wikis / keep their content to a minimum and always link to that one documentation. So when it gets updated, no matter where someone initially searches, they always get the full up-to-date information.

[Korne127]

I have another update: I finished rewriting the updating information page: Improving the page structure and all explanations (making them more concise and easier to follow), correcting wrong information that didn't work and adding missing information, giving a better overview about different approaches and their (dis-)advantages and replacing information that depends on a specific project structure with information that's generally applicable.
I also added a corresponding section in the user guide with the basic information a user should know.

[Korne127]

Another bulk of changes I've spent the last week working on :D

It's hard to summarise the changes because it's just so much, but I still tried to do it. There are many improvements that aren't included in the following summary; the individual commit messages are much more complete and detailed.

Major improvements of these new changes:

• The whole reference section has been restructured and improved:
  • Content that is directed to users or application authors have been merged into the guides or removed when duplicated.
  • All reference information about AppImages has been rewritten and bundled into a new AppImage reference section. This way, the specification and reference implementation decisions could be separated much more clearly and many duplicates could be removed.
  • Much information (such as about different AppImage versions, their differences and history) has also been added.
  • The whole reference section has been updated accordingly to the change to the new static runtime being the official reference implementation and AppImageKit being split up into different repositories.
• The introductory pages have been rewritten and heavily improved:
  • The front page now directly links to the chapters that people in different situations should read, and makes it much easier to find important information most people might search for.
  • The quickstart and the FAQ have been merged to have a single page to point new users towards, which answers all common questions.
  • The introductory chapter pages have been improved to give a better summary of their content.
• The AppImageHub section has been rewritten to a general software catalog section, giving an overview on what a software catalog is and why one would want to use them and listing the different software catalogs. A corresponding downloads section has been added to the user guide.

Other pages that have been rewritten:

• The signature page has been improved, with more concise explanations and an overview as well as information on how to use signatures with the individual tools.
• The AppStream page has been improved, with a new validating section, advantages and a new explanation on how to embed AppStream files with AppImage creation tools that create the AppDir.

General improvements that have been done to the documentation:

• Many pages have been restructured with an improved order and a better explanation and overview on what the page actually explains.
• Duplicated information has been removed and explanations have been made easier to understand with more examples.
• Outdated information, e.g. about the discourse forum, has been removed.
• Wrong links and references have been corrected.

[Korne127]

@probonopd I am a bit confused about your comment in AppImage/AppImageKit#877 (comment).

As I've worked my way through the entire AppImage project while updating this documentation (being it the AppImage creation tools, the updating system, software catalogs or now the history and different types of AppImages), I've made some issues, comments or pull requests on respective repositories. I usually wrote that I stumbled across that while rewriting the documentation.

But now you've stated that I should make an issue before working on the documentation, implying you are not aware of my basically complete rewrite of it (which is really necessary, see the motivation I wrote when I made this PR).
So yeah, I am a bit confused, and as I've spent several months working on this (and far over a hundred working hours), of course also a bit anxious you might have different plans.

Therefore I wanted to ping you again and hurried the last week to finish this batch of changes (as this PR is now mostly done, there are only many minor changes and a few pages I want to majorly improve left).

The work required to rewrite the documentation to be consistently up-to-date without duplications with clearly separated chapters so that every specific information is easy to find and intuitive and concise explanations and information is insane, so I really hope that you can see and appreciate it.

[Korne127]

Another smaller update; I finally finished going through every page once. There are still some things on my to do list, but most are either smaller consistency changes or things I already wrote a todo for.

Those are the new changes:

• The testing page has been majorly improved, updated and corrected.
  • The testappimage part has removed (as it's outdated and doesn't work with modern AppImages) and replaced by an explanation on how to manually test with live images.
  • The appimage-testsuite part has been removed (as it's outdated and doesn't work out of the box) and replaced by a new script which is an improved version (which works, is not outdated, future-proof and consists of less code) with explanations.
  • The descriptions and explanations on the page have also been improved.
• Information about the desktop integration tools Gear Lever, zap, Soar and AM has been added to the desktop integration page in the user guide. The other tools' information has also been improved.
• The pkg2appimage page has generally been improved to not have duplicated sections and to have easier to understand explanations and updated to use the prepackaged AppImage.
• The FUSE troubleshooting page has been updated to account for the fact that new static runtime is now the official reference implementation runtime.
• The troubleshooting page has been improved to contain the information on how to see potential error messages.