automatic bundling of dylibs -- Xcode template step or addon solution?

[artificiel]

I'm working on a project where I need to install a macOS app on a bunch of machines that I don't want to burden with homebrew/Xcode. the app makes uses of .dylibs that are not found when copying the .app over. this is fixable in script by rsyncing the dylibs within the .app, and doing the otool/install_name_tool thing, but it is brittle and pretty tedious (and does not survive a project generator pass).

dylibbundler is a utility that scans a binary for paths (and recursively scans the found dependencies too), ignores the system stuff, copies the rest, and does the install_name_tool dance to link things. it also signs.

I made a final build script phase that parses the OTHER_LDFLAGS (which is set by addons to point to their eventual dylibs) to help the bundler find the dependencies, and it "just works" -- when the processed app is thrown on a virgin machine with a similar version of macOS it just needs the right-click gatekeeper "open" action and otherwise runs.

There are sometimes forum questions on how to manage such an integration of dependencies and I don't think there is a simpler way, so I find it would a useful addition to the Xcode template BUT it relies on another tool... not sure what to make of that. The software in question seems mature and kept updated (I tried a few other scripts but they are not as powerful (and did not work as seamlessly)).

Having to need to turn on OF_BUNDLE_DYLIBS=1 switch in the project.xcconfig means casual users would not encounter the step unless they try to, at which point a few lines of explanation should suffice. The tool is available in homebrew, so within quick reach for someone who wants to deploy a self-contained app.

The true problem is that project generator wipes out tweaks like that -- if they were conserved it would not be too much of a problem to set up something like that once manually. If we want to keep the Xcode template as lean as possible perhaps the real solution is to provide a way to re-write a project taking into account certain directives? I don't know enough about PG to have a clear opinion on that -- maybe a 2nd little Xcode that gets "merged" into the main one, but not squashed if existing, on re-generation? or maybe allow an addon to post-process the Xcode project? ofxBundleDylibs would then re-insert the script during project generation? (that could be useful for other things).

In any case here's the script (which can probably be revised/simplified):

# run as zsh
flags=(${(@s: :)OTHER_LDFLAGS})
declare -A paths
declare -s sargs

# use the keys of an associative array to accumulate unique directories enclosing .dylibs based on OTHER_LDFLAGS
for ITEM in $flags; do
    if [[ $ITEM == *'.dylib' ]] then
        d=$(dirname "$ITEM")
        paths[$d]=1
    fi
done
    
# construct a list of -s args for dylibbuilder to find the dylibs
for key val in "${(@kv) paths}"; do
    sargs+=-s\ $key\ 
done

# do the thing
/opt/homebrew/bin/dylibbundler -cd -b -x "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME" -d "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/libs" ${(s[ ])sargs}

looks like this :

[dimitre]

That's super useful. we can add the script in XCode project itself, so it will survive projectGenerator.

[ofZach]

one note is that it's possible to have templates with the PG, so if you have something custom in your Xcode project, it will survive the PG tool. It may be nice to have something like a guide for the PG with steps about how to do this. I think writing up notes about how to do this may be really helpful.

The pg itself is destructive to settings, because it always starts with an empty "of" style project that doesn't have custom settings and it doesn't really have any smarts about parsing an existing project. it just looks at things like addons.make and the src folder, to figure out what to add to an empty project.

I've created templates before when an add-on requires a bunch of work -- an example would be ARKit -- that way I don't need to redo all that work when I make new projects that use ARKit but want to try adding opencv or some other add-on, etc. the same thing could make sense if you have a specific build step or something specific to the Xcode project file.

[artificiel]

ah! yes the templates. my concern with that is that it creates a kind of "external dependency" for the project. where do you commit that template, and is there a way for PG to automatically load it if available? (note that the script above is project-agnostic; all required params are extracted from the project itself, so it's not tied to a specific config/use case)

could it be possible to have a "preprocess/postprocess" scripting step to the addons ex:

osx:
   ADDON_POSTPROCESS=scripts/xcode_add_bundle_step.sh

it would add some power to addons, for instance to manage dependencies that are not provided as binaries within the addon but obtainable through the platform package managers. it could also be used to synchronize assets (for instance a specific version of an addon downloads a specific version of a dependency, code or other (ex: ML networks)).

and (forwarding the project build env variables) for my current usage I could use cocoapods' ruby gem to manipulate the xcodeproj (there seems to be a python equivalent) so I could make an addon to contain that functionality (and make it a dependency of the project) -- if the PG could trigger a script hook.

I like the idea of leveraging addons as "generic project configurators", but maybe there's a simpler way, for instance predetermining a name like project_post_process.sh that gets executed by PG if present in a project folder?

I guess a preliminary question is: is it preferable to provide a solution to the specific problem of bundling dylibs as an integrated feature (where simply adding the step and OF_BUNDLE_DYLIBS=1 with instructions to run brew install dylibbundler is the simplest path) at the expense of making the Xcode template a bit more complex (greater surface to support), or is it more interesting to make it so that PG can be "scripted" (thus enabling the possibility of automatically bundling dylibs, but other things as well).

[artificiel]

after a test I confirm that the ruby lib approach works, so if we can get something like this to execute post-PG, it solves the problem without modifying the Xcode template (assuming gems are installed). I'm still unsure about what is the preferable approach.

append_dylib_bundle_step.rb:

require 'xcodeproj'
path_to_project = "/path/to/openFrameworks/apps/myApps/depthAiTracker/depthAiTracker.xcodeproj"
project = Xcodeproj::Project.open(path_to_project)
main_target = project.targets.first
phase = main_target.new_shell_script_build_phase("Bundle dylibs")
phase.shell_script = "/bin/zsh /path/to/openFrameworks/scripts/append_dylib_bundle_step.sh"
project.save()

[dimitre]

About project generator wiping configurations like this

OF_BUNDLE_DYLIBS=1 

I've added an optional include #include? "App.xcconfig" in Project.xcconfig which changes nothing in normal projects.
The question mark means it won't complain if the file is not there.
but I can keep customizations in a separate file, like custom icons, project version, etc so it survives projectGenerator.

I'll be happy to try a script build phase to check if it works well here (ofxNDI, ofxHapPlayer are two useful for me)

[artificiel]

ah! good to know about App.xcconfig!

here is what I have as the last build phase. it takes a few seconds. note that it's a /bin/zsh, not a plain /bin/sh. I fiddled a bit with the dylibbundler flags to prevent re-copies and re-processing of the paths; it seems the flags works well in the OF workflow both for a clean project, and for iterative re-compiling.

# set shell to /bin/zsh
# requires `brew install dylibbundler` (v1.0.5 as of 20230121)

flags=(${(@s: :)OTHER_LDFLAGS})
declare -A paths
sargs=""

# use the keys of an associative array to accumulate unique directories enclosing .dylibs based on OTHER_LDFLAGS
for ITEM in $flags; do
    if [[ $ITEM == *'.dylib' ]] then
        d=$(dirname "$ITEM")
        paths[$d]=$d
    fi
done
    
# construct a list of -s args for dylibbuilder to find the dylibs
for key val in "${(@kv)paths}"; do
    sargs+=-s\ $key\ 
done

# do the thing
/opt/homebrew/bin/dylibbundler -cd -b -x "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME" -d "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/libs" ${(s[ ])sargs}