Make swift-doc available on Windows

[compnerd]

I thought I would just create a tracking issue in case others are interested. I think that this can also serve as an easy communication point as I'm working through the various libraries.

Known Items to Handle

• bump swift-log dependency for Windows support (Update swift-log dependency #248)
• bump swift-argument-parser dependency in SwiftSyntaxHighlighter for Windows (build: bump swift-argument-parser to 0.3.2 NSHipster/SwiftSyntaxHighlighter#14)
• add support for Swift 5.4, 5.5 to SwiftSyntaxHighlighter (Add version-specific manifests for Swift 5.4 and 5.5 NSHipster/SwiftSyntaxHighlighter#15)
• update SwiftSemantics for Swift 5.4+ (SwiftSemantics: update keyword SwiftSemantics#12)
• fix #import usage in libxml2 module handling in Markup (Update libxml2-markup.h Markup#2)
• handle CLong to Int conversion in Markup (Update Node.swift Markup#3)
• silence warning from pkg-config handling in Markup (Update Package.swift Markup#4)
• bump SwiftSyntaxHighlighter dependency in swift-doc for Windows support (Add support for Swift 5.4 / Xcode 12.5 #252)
• bump SwiftSemantics dependency in swift-doc for Windows support (Add support for Swift 5.4 / Xcode 12.5 #252)
• repair the Windows build of Markup (build: repair the clean build on Windows Markup#5)
• silence warning from pkg-config handling in GraphViz (build: silence an annoying warning on Windows GraphViz#20)
• handle libxml2 linkage across platforms for Markup (Modules: repair libxml2 link on Windows Markup#6)
• bump swift-log dependency in swift-doc (Update swift-log dependency #248)
• fix Foundation usage for Windows in swift-doc (swift-doc: load FoundationNetworking whenever possible #265)
• bump swift-argument-parser dependency in swift-doc for Windows support (build: bump swift-argument-parser dependency #259)
• build GraphViz for Windows
  • fix gvToolTred export (https://gitlab.com/graphviz/graphviz/-/merge_requests/1925)
  • handle modular build issue with GraphViz and WinSDK module (https://gitlab.com/graphviz/graphviz/-/merge_requests/1930)
• add Windows CI coverage for CommonMark (CI: add Windows coverage CommonMark#32)
• add Windows CI coverage for SwiftMarkup (CI: add Windows coverage SwiftMarkup#14)
• add Windows CI coverage for Markup (CI: add CI job for Windows Markup#8)
• repair apple/swift-syntax 5.4 for Windows (build: use Foundation instead of C for getenv apple/swift-syntax#283)
• add easily accessible builds of GraphViz for Windows 🚧
• track the 5.4 branch of swift-syntax in SwiftSemantics for Windows support (Update [email protected] SwiftSemantics#16)
• bump Markup dependency in swift-doc for Windows support (build: bump Markup to 0.1.2 #268)
• add Windows CI coverage for GraphViz (CI: add Windows test coverage GraphViz#24)
• re-enable transitive reduction support in GraphViz (Revert "Conditionally compile call to gvToolTred on Windows" GraphViz#25)
• add Windows CI coverage for SwiftSemantics (CI: add Windows CI SwiftSemantics#15)
• bump GraphViz dependency in swift-doc for Windows support (build: bump GraphViz dependency to 0.4.1 #274)
• handle invalid file name generation with swift-doc (: is not portable) (perform some path sanitization over the generated names #258)
• get prebuilt binaries for Windows
• fix dependency resolution (Update [email protected] #278)
• add Windows CI coverage for swift-doc (Update ci.yml #279)
• package swift-doc for Windows (build: add a GHA for packaging swift-doc on Windows #280)
• figure out GitHub actions handling for Windows packages

[compnerd]

@mattt, first of all thanks for the tooling here, it really is very nice. I CC'ed you here since I think you may be interested in this work and I could use a bit of help with some of the items that need to be sorted out.

Although there are a lot of small items still on the list (and I'm slowly trying to work through them), the resulting work is promising. There is a lot of clean up work to be done still, but I am able to generate documentation on Windows for Windows packages now. I have some intermediate content that I've thrown up while testing and working through the necessary automation to get this usable. You can see the results (for now at least) at https://compnerd.github.io/swift-win32/SwiftWin32/.

There are some problems that I've noticed in the results which to me is unclear if it is Windows specific or not.
1. The inheritance diagram seems to be right shifted. An example is at https://compnerd.github.io/swift-win32/SwiftWin32/ProgressView/. EDIT: this was related to an issue with foundation usage
2. I cannot tell what the actual behaviour is supposed to be for the list of documented types, which currently appear to be unordered. This is likely a non-Windows specific issue, but Id like to make sure that it is the case.
3. It seems that some of the HTML pages do not link properly, at least when served via GitHub pages. This normally applies to _ prefixed types/names which are technically not meant to be public, but do get listed current. This is somewhat related to #232. EDIT: this is related to jeykll on GitHub, which probably deserves some documentation.

[compnerd]

@mattt would you mind rolling a new release for SwiftDocOrg/GraphViz to allow us to bump the minimum version? I think that with everything in flight, we are nearly ready for getting swift-doc usable on Windows for general consumption.

[compnerd]

@mattt btw, if you are interested in the current state of things with this effort. With #276 I can build swift-doc for windows (pre-built snapshot: https://github.com/compnerd/swift-build/releases/tag/swift-doc-5.5-DEVELOPMENT-SNAPSHOT-2021-05-02-a) and can actually run this on Windows to generate things: https://compnerd.github.io/swift-win32/SwiftWin32/.

Usability isn't entirely great yet: https://github.com/compnerd/swift-win32/blob/main/.github/workflows/documentation.yml due to the distribution of the binaries. I'll try to see if I can whip up some WiX rules to generate a MSI with the necessary dependencies packaged in. That would allow the installation to be much simpler. The rest of the complexity would come down to the deployment which is somewhat more specific to me (I wanted a single commit branch with the results).

[compnerd]

@mattt any opinions on how to handle the distribution aspect of this? Would you prefer a just a zip with the necessary dependencies or a MSI? If its the MSI, any thoughts on locations for installation and where to place the files for the MSI generation?

In terms of licensing, I believe that expat is MIT/BSD, and graphviz is Eclipse Public License. Since we are using upstream repositories for the builds of those dependencies, I think that just indicating the revisions for the repositories should be sufficient. expat is an implicit dependency and used by graphviz. I want to test the packaging since Id like to remove a few of the "plugins" so that we don't need to worry about pango and cairo dependencies.

[mattt]

@compnerd I'm unfamiliar with how Windows Installer works, so apologies for asking some really basic questions:

• Are .msi files self-contained? That is, can a user install swift-doc by downloading and opening one?
• Could we build it using GitHub Actions in a separate release job?

However we decide to package it, we can host any artifacts on GitHub alongside our release — just as we do for our Homebrew bottles.

[compnerd]

Well, it would be semi-self-contained. Yes, a user would be able to download and "open" one to install. But the expectation would be that they would be responsible for installing the dependencies themselves - namely, the Swift toolchain.

Id like to experiment with building the MSI via WiX and MSBuild as GitHub actions. We could integrate that into the release job.

[mattt]

Well, it would be semi-self-contained. Yes, a user would be able to download and "open" one to install. But the expectation would be that they would be responsible for installing the dependencies themselves - namely, the Swift toolchain.

That sounds entirely reasonable to me! The important part is that users don't have to navigate installing the other system dependencies (libxml2 and graphviz).

Id like to experiment with building the MSI via WiX and MSBuild as GitHub actions. We could integrate that into the release job.

Yeah, GitHub Actions should give you everything that you need. The only downside is that it can be difficult to debug workflows. Let me know if you have any questions about getting that to work 😃

[compnerd]

@mattt btw, I was wondering if there was a codesigning certificate for swiftdoc.org? That would be really nice to have access to as a GHA secret to be able to sign the generated MSI.

[compnerd]

@mattt @MaxDesiatov - not sure who normally cuts the release, but it would be nice to get a pre-release MSI for swift-doc to start experimenting with it.

[mattt]

@compnerd That's me. Apologies again for not being able to pay more attention to swift-doc last month. My schedule is open this week, and I'm working on the next release (RC1) now.

[mattt]

@compnerd Do we have any release notes / instructions for how to use swift-doc on Windows? I'm cutting a release now, and would love to have more information for folks to try it out themselves.

[compnerd]

I can try to write some up; it largely will come down to:

• run msi (install)
• start new shell
• swift doc --help

I do have one better though. Once we have a URL for the MSI, I intend to have a pre-configured GHA at https://github.com/compnerd/swift-win32-application to demonstrate the setup :)

[mattt]

@compnerd That'd be really helpful. Thanks! Doesn't have to be anything fancy. If you give me the exact commands, I'd be happy to fill out the prose.

[compnerd]

Well, that's really the beauty of what we managed to get done with the release :) The entire reason for the MSI was to actually avoid the commands. It is dependent on your approach, from the GUI its just double click on the MSI and click continue and okay when it finishes. From the console, you can just run the installer in quiet mode .\swift-doc.msi /quiet IIRC. Then just launch a new shell (it goes through some effort to setup everything, including additions to Path) and start using it!

[compnerd]

Ugh, the build failed for a different reason:

D:\a\swift-doc\swift-doc\WiX\swift-doc.wxs(3): error CNDL0108: The Product/@Version attribute's value, '1.0.0-rc.1', is not a valid version.  Legal version values should look like 'x.x.x.x' where x is an integer from 0 to 65534. [D:\a\swift-doc\swift-doc\WiX\swift-doc.wixproj]

How do you want to handle this?

[mattt]

@compnerd Do you have a working artifact that you built locally? For now, we could upload that as a release artifact for 1.0.0-rc.1. You should have the necessary permissions to do that. If not, I can take care of that.

[compnerd]

@mattt - oh, thats a good idea. I don't have the artifact, but its easy to get, cause you built it already! https://github.com/SwiftDocOrg/swift-doc/runs/2723178219 has the MSI!

[mattt]

@compnerd Perfect! 1.0.0-rc.1 now has swift-doc-1.0.0-rc.1.msi as a release asset. Does that file work as-is? Like, should we host a Zip file instead? Or are there any problems using that file name?

[compnerd]

I really would prefer to leave it as is. The file works as is and this really does help simplify usage in projects as well (for deployment on GitHub Actions).

[compnerd]

Bleh, correction - it almost works. It seems that something is not entirely correct. A previous build (https://github.com/compnerd/swift-build/runs/2551131399?check_suite_focus=true) seems to work, however, the version from the MSI doesn't (https://github.com/compnerd/swift-win32/actions/runs/929619825) ...

Error: The loaded '_InternalSwiftSyntaxParser' library is from a toolchain that is not compatible with this version of SwiftSyntax
Error: The loaded '_InternalSwiftSyntaxParser' library is from a toolchain that is not compatible with this version of SwiftSyntax
Error: The loaded '_InternalSwiftSyntaxParser' library is from a toolchain that is not compatible with this version of SwiftSyntax

Clearly not a packaging/file issue, but rather a version mismatch. What is interesting is that the toolchain is exactly the same version - 5.5-DEVELOPMENT-SNAPSHOT-2021-05-09-a.