Make geerlingguy/drupal-vm Docker image easy to use for single-site installations

[geerlingguy]

Issue Type

• Feature Idea

Summary

In #1356, I built a base drupal-vm Docker image, and so far it does a lot of things nicely, but there are a few issues that prevent me from saying "yes, this is a good start". I want to fix those issues before I start actually advertising the base image / Docker image build functionality from #1356. Namely:

• The php7.1-fpm service is dead when you docker-compose up. (Workaround for now is to run: docker exec drupal-vm service php7.1-fpm restart)
• The default vhost (drupalvm.dev) is embedded in the image and isn't able to be changed unless you do an ansible-provision (really heavyweight), or log in and manually swap it out yourself.
• Ditto for the location inside the VM where Drupal VM expects a Drupal codebase (currently it's /var/www/drupalvm/drupal/web—kinda verbose).
• I need to document a real-world example use case (e.g. I have a Drupal codebase, and I put the docker-compose.yml file inside, then I ran commands X, Y, and Z and it was working...). I have it working with a couple of my small D8 sites right now, but it's a little janky.
• As part of the above, document how you can get MySQL's data dir to persist.

All in all, it performs pretty well so far. Maybe I could wrap a few shell scripts inside the container so users could use like docker exec drupal-vm drush cr or something like that (and then power users could alias the docker exec drupal-vm part for ease of use).

[geerlingguy]

For the first issue, with FPM not running on container start:

root@128d94d20d15:/var/log/mysql# service php7.1-fpm status
● php7.1-fpm.service - The PHP 7.1 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.1-fpm.service; disabled)
   Active: inactive (dead)
     Docs: man:php-fpm7.1(8)

For some reason it's not set as enabled. Going to boot up a fresh Drupal VM instance on Ubuntu to confirm whether it's enabled or disabled by default there... maybe it's a Debian issue?

[geerlingguy]

Same thing inside a Vagrant/VirtualBox VM with the Debian 8 base box:

$ systemctl status php7.1-fpm
● php7.1-fpm.service - The PHP 7.1 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.1-fpm.service; disabled)
   Active: active (running) since Mon 2017-05-15 03:20:00 UTC; 26min ago
     Docs: man:php-fpm7.1(8)
 Main PID: 6170 (php-fpm7.1)
   Status: "Processes active: 0, idle: 5, Requests: 0, slow: 0, Traffic: 0req/sec"
   CGroup: /system.slice/php7.1-fpm.service
           ├─6170 php-fpm: master process (/etc/php/7.1/fpm/php-fpm.conf)
           ├─6171 php-fpm: pool www
           ├─6172 php-fpm: pool www
           ├─6173 php-fpm: pool www
           ├─6174 php-fpm: pool www
           └─6175 php-fpm: pool www

Testing on Ubuntu 16.04:

vagrant@drupalvm:~$ sudo systemctl status php7.1-fpm
* php7.1-fpm.service - The PHP 7.1 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.1-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-05-15 03:53:40 UTC; 1min 22s ago
     Docs: man:php-fpm7.1(8)
 Main PID: 32566 (php-fpm7.1)
   Status: "Processes active: 0, idle: 5, Requests: 0, slow: 0, Traffic: 0req/sec"
   CGroup: /system.slice/php7.1-fpm.service
           |-32566 php-fpm: master process (/etc/php/7.1/fpm/php-fpm.conf)                      
           |-32569 php-fpm: pool www                                                            
           |-32570 php-fpm: pool www                                                            
           |-32571 php-fpm: pool www                                                            
           |-32572 php-fpm: pool www                                                            
           `-32573 php-fpm: pool www                                                            

May 15 03:53:40 drupalvm systemd[1]: Stopped The PHP 7.1 FastCGI Process Manager.
May 15 03:53:40 drupalvm systemd[1]: Starting The PHP 7.1 FastCGI Process Manager...
May 15 03:53:40 drupalvm systemd[1]: Started The PHP 7.1 FastCGI Process Manager.

So it is enabled on Ubuntu 16.04. But not on Debian 8. Ah, systemd...

I can't pin it on being the geerlingguy.php role's fault, though—the task as written should be working, maybe it's an issue in my Docker container's setup... I'll have to run the tests in my role's test container and see if it's the same there.

[derimagia]

Not sure if you're still looking at this but I was curious to see what was done. In case you haven't seen it it looks to be that there's no entrypoint and the cmd is just bash. The systemd service isn't running since there's no entrypoint and the cmd is bash so you're just pretty much starting the process manually.

[geerlingguy]

@derimagia - In this image's case, the docker-compose.yml example would set the entrypoint to launch systemd, which inside the container manages the starting of any configured services (which are enabled), and in the case of php7.1-fpm on Debian 8, it's not enabled for some reason—all the other enabled services come up fine.

I'm working on that issue (which seems to not be exclusive to Drupal VM) over in geerlingguy/ansible-role-php#191 — Ansible's service module seems to work fine on Fedora, Ubuntu, and CentOS with systemd... but something is failing silently on Debian 8, at least with PHP-FPM from Ondrej's apt repository.

[derimagia]

I see, I'm not used to those being flipped (shell being default, having to overwrite the process) but that's my fault for finding the compose. I know you're trying to do MVP now and if there's anything that needs to be done there let me know. Is running using systemd a permanent solution or was there a thought to eventually separate it out?

[derimagia]

Also making a note since I didn't see it above, there were logs in the docker file for when it was created, we should probably clean those up (php-fpm said it started 2-3 days before I started the container)

[geerlingguy]

@derimagia - Yeah, the intention right now is just to get Drupal VM 'tricked' into thinking a Docker container is a normal VM, using systemd or sysvinit to manage processes.

At some point in the future, components of Drupal VM will be parceled out and put in separate containers—however, I am not planning on doing all of that until the point where the end-to-end UX of Docker gets nearer the UX of a tool like Vagrant—and without me building (and having to maintain) layers upon layers of all the shims needed to do it today. (For example, I'm not interested in building a UI like Kalabox, or shipping everything inside VirtualBox using Docker Machine. If I start using Docker for Drupal VM, by golly, I'm not going to wrap Docker inside even more layers of virtualization!

For an example of one of the images I've started playing around with separately, see docker-image-solr. But note that I am still not following what I believe is the mainline/conventional container approach there, as I'm working to make the transitional state from full VMs to containers more sustainable (e.g. I don't want people building ugly-as-all-get-out shell scripts inside Dockerfiles inside other shell scripts to do basic configuration management). Hopefully ansible-container bridges that gap... it's improved a lot in the last year, though I'm still not using it anywhere publicly.

[derimagia]

Makes sense, a little disappointing but I understand not having time for it. I'll need to mess around with this. I know a lot of it is a bit different from what I've seen ansible being used for. One thing that would need to be done would be to move a lot of the configuration into the entrypoint as opposed to baked in, so the ansible scripts would essentially need to separate out the "installation" part and the "configuration" part.

Although I'm not sure if docker will ever evolve into what you're looking for exactly. For better or worse Docker doesn't really have any tools to help provision / create docker images, it just gives you lower level tools for it. docker-compose helps if you already have the images provisioned obviously but that's about it. They leave it out to third parties to make these (Which makes sense from their perspective I think) but makes it harder for people to learn how to actually do things like this. I do agree docker's UX is to be desired.

If you didn't see yet, https://github.com/moby/moby/releases/tag/v17.05.0-ce adds multi-build support which will definitely help with what you want. It's actually pretty neat.

[geerlingguy]

One way to solve the virtualhost issue could be to integrate with https://github.com/jwilder/nginx-proxy — it's still hackier than I'd like, though. Throwing an additional proxy in front of Apache (or potentially Nginx) is a recipe for pain, in terms of little things that could go wrong with the request routing from host to nginx to container (for things like bigpipe, varnish, streaming requests, etc.).

[geerlingguy]

Another possibility could be adding something like gotpl, which seems to follow Jinja2/Twig syntax in some ways, and would allow environment variable substitution similar to what Wodby does: https://github.com/wodby/nginx/blob/master/1.10/Dockerfile#L45

I just don't want to go too far down that route and diverge the Docker configuration setup from the Drupal VM proper configuration setup.

[geerlingguy]

Grr... that first issue might not yet be solved...

root@5d04d77b57ce:/# service php7.1-fpm status
● php7.1-fpm.service - The PHP 7.1 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.1-fpm.service; disabled)
   Active: inactive (dead)
     Docs: man:php-fpm7.1(8)

[derimagia]

Not sure if this is a problem but to help simplify config the ansible provision for building the container there could be 2 (mental) types of configuration - the ones that are configurable by drupalvm and the ones which are not. It changes some of the values and the other ones are left alone (or slightly modified if need be). Then on the entrypoint it just runs the templater (e.g. gotpl) to replace the remaining values.

[geerlingguy]

Strangely, using the new composer docker-bake command, then saving the image (composer docker-save) results in an image that does, indeed, have php7.1-fpm running and enabled inside Debian 8. So there's some difference between the Dockerfile and the bake.sh script.

[thom8]

Yeah, there's a big difference between docker build & docker run.

build will not retain any state, even between commands whereas run will.

So for example you can't set an environment var in build -- export VAR1=test then use it in the next command, but you can with run.

Same goes for running services.

[geerlingguy]

I think I might be hitting moby/moby#1916 :/

[derimagia]

Hmm, honestly you shouldn't ever really need privilege in build, what's the reasoning it needs it?

[geerlingguy]

@derimagia a lot of things, like iptables configuration, requires certain privileges... if you're looking at Docker from the perspective of a crazy person who wants to make Docker work like a VM, then it's nice to be able to have privileged access :)

I'm not saying that's a good thing, mind you (though I don't consider it as bad and evil as Docker purists would paint it)... just that it makes things work identically between a full fledged VM and a Docker container. There's a reason why that issue has been open for 4+ years and has ~300 comments—it's a #dockerWTF imo that you can do more with docker run than docker build.

[derimagia]

Having that would mess with portability, like you're doing right now. Running iptables goes past userspace and you're messing with things that expand it. Even if docker build supported the flag you would need to run it on each invocation of the container when running it, otherwise it wouldn't persist anyway.

[geerlingguy]

Having that would mess with portability, like you're doing right now.

Somewhat... right now my solution is to disable the security/networking modifications Drupal VM would normally make, since it's assumed Docker will handle things like container networking.

[geerlingguy]

I just got my local test site running and happy during a Drupal meetup tonight, so at least there's that—just pushed a commit which gets FPM to actually be running on container start, yay!

But this is definitely going to be 'experimental' for some time. It's a little clunky... but it's nice enough that I've already switched one of my other local sites from using Drupal VM with Vagrant to using a Dockerfile and docker-compose.yml, and not having any Drupal VM integration besides :)

[geerlingguy]

Also reading through https://www.vagrantup.com/docs/docker/configuration.html ... could be possible to make the bridge to native Docker easier with Vagrant to layer itself between things. I'll see if it could be simpler/easier to go that route (even though one would need both Docker and Vagrant in this scenario).

[geerlingguy]

I might want to split out into a separate issue the work of trying to use Docker + Vagrant, though... @thom8 what do you think? Since we're in the 'experimental' phase, I'm happy to work on a few different angles (but want to get it working with just Docker first, as a POC).

[thom8]

Yeah probably a good idea to split that out, I'd like to get to a stage where Docker was a drop in replacement for vbox in the current stack so it can be enabled in existing config.

But still think it would be best to get a Docker only solution stable then look at adding Vagrant support for backwards compatibility.

It's also important for advanced use cases to make sure it all works standalone.

[geerlingguy]

@thom8 - Opened #1381 as a follow-up issue.

[geerlingguy]

For the last two items:

• The Apache vhosts configuration for Docker
• How to get the MySQL data dir to persist

I'll leave them off for now, in order to get the already-working stuff 'shipped' in 4.5.0.

I've been using the Docker setups described in the updated docs for a few of my sites locally the past couple weeks, and those two issues haven't been as big of annoyances as I thought they'd be, so I figure we can wait for others to highlight things that aren't working great or could be low-hanging fruit.