Pulling between different folder structures

[LucasDemea]

Describe the bug
I'm a long time wordmove user, and today I'm facing an issue that I can't understand.

I'm trying to pull a site that has a different folder structure than my local site (local : bedrock ← production : vanilla WP). I specify the custom paths in each section to match the file structure of both sites, but the files are pulled in the same folders as the source site instead of going into the destination site custom folder structure.

My final goal is to convert a vanilla WP install to a Bedrock WP structure.

EDIT This occurs only when using ssh protocol, but works as expected when using ftp.

Wordmove command
wordmove pull -e production -u

Expected behavior
The files should be downloaded according to the folder structure defined in my local section.

movefile.yml

local:
  ...
  paths:
    wp_content: "app"
    uploads: "app/uploads"
    plugins: "app/plugins"
    mu_plugins: "app/mu-plugins"
    themes: "app/themes"
    languages: "app/languages"
  ...

production:
  ...
  paths:
    wp_content: "wp-content"
    uploads: "wp-content/uploads"
    plugins: "wp-content/plugins"
    mu_plugins: "wp-content/mu-plugins"
    themes: "wp-content/themes"
    languages: "wp-content/languages"

Exception/trace
No errors

Environment (please complete the following information):

• OS: Ubuntu 20.04 VM
• Ruby: ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-linux-gnu]
• Wordmove: 5.2.1

Doctor

• wordmove doctor is all green

▬▬ Validating movefile section: local ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

    ❌  error | [/database/port] '3306': not a string.

I changed the movefile to fix the error and got all green, but it didn't fix the issue.

[LucasDemea]

I just found out that this occurs when using ssh protocol, but it works as expected using ftp.

[alessandro-fazzi]

Really cozy report! Thank you for the effort and for the report itself.

I'll certainly force my schedule to take the time to look at this.

[LucasDemea]

Anything new on this ?

[alessandro-fazzi]

While working on the new release, I wanted to investigate about this one and possibly remove a bug before going stable with 6.0 release.

Unfortunately it was - and remains - a complex journey. And it would require a pamphlet in order to explain all the historical layers laying beneath.

Let's try to have a rationale at least.

First thing first: as it stands, wordmove does not support a different folder structure in local than in remote when using rsync+ssh.

This is not documented and neither is documented the possibility to do such an operation; but I agree that it would be the expectation.

I have reasons to think that this became a fact starting with 1642065, commit that changed a lot - too much I'd say - about the rsync behaviour we've behind the scenes, but fixed what I considered a bug.

What was the problem which brought us to implement such a big change?

The exclude section of movefile.yml never worked consistently before that commit. For example, with a structure

/
|_wp-content/
  |_plugins/
    |_foo.php

and in movefile.yml

exclude:
  - wp-content/plugins/foo.php

the exclusion wouldn't have been honored, because the rsync command was transferring /rootpath/wp-content/plugins folder and rsync's exlusion logic considers exclusion path as "relative" (quoted because rsync's "relative" concept would require another pamphlet) to the transferred folder. Bringing to:

Do /rootpath/wp-content/plugins/wp-content/plugins/foo.php exists? No, so nothinig to exclude here!

I thus needed to think a new approach in order to be able to compile exclusion list as expected and have rsync honoring what the user meant.

What is the new approach doing?

The very very simplified version is: wordmove tells rsync to always sync the wordpress_directory (the root of your project/wordpress), to exclude everything inside the wp-content directory (keeping in consideration custom paths) BUT specifically include wp-content/plugins/. This way the natural rsync logic develops like

Being syncinc the folder /rootpath/, does /rootpath/wp-content/plugins/foo.php exists? Hell yeah! Exclude it all the way!

Sounds good, but why we should care about this thing?

There is a counter effect emerging from this behaviour, and it's what makes actually impossible to manage different folder structure through rsync+ssh. We're telling rsync to sync the wp-content/plugins/ folder inside /rootpath/ folder and it can only transfer it as it is: rsync cannot rename things while transferring. the result is that we're ignoring wordmove's local.paths config while pulling and remote.paths config while pushing.

The result is that only symmetric dirtrees are actually supported.

So what?

I thought about what else I could invent to take all the pieces together and the only approach I could think about is this one:

• put back the "old" rsync behaviour
• I think alos the old behavious would partially fail in managing the proposed scenario, so:
• update the "old" behaviour telling rsync to not sync our folders, but contents of our folders, in order to enable wordmove to control folder structure
• update the logic behind wormdove's exclude list: we'll have to programmatically read it and transform exclusion paths for each folder we're pushing/pulling. Example follows:

Given a movefile

local:
  paths:
    wp-content: app/
    plugins: app/plugins/

production:
  exclude:
    - wp-content/plugins/my_plugin/
    - /something.php

and a remote dirtree

/remoterootpath
|_something.php
|_wp-content/
  |_plugins/
    |_my_plugin/

and a local dirtree

/rootpath
|_app/
  |_plugins/
    |_my_plugin/

When the user wordmove pull -w we can take the plain exlude list as configured by the user and the simplified rsync command would be rsync production:/remoterootpath/ /rootpath --exclude wp-content/

When the user wordmove pull -p we have to programmatically transorm the exlude list like this

- my_plugin/

• removing wp-content/plugins/ because we're working "inside" that folder and we need to relativize what the user meant
• removing the esclusion starting with / becuse they were intended for the "root" of the project, not for the "root" used while transferring the plugins.

The simplified rsync command would be rsync production:/remote/rootpapth/wp-content/plugins/ /rootpath/app/plugins (note the closing / meaning "the contents of")

The missing thing from this picture is the programmatic transforming log while interpreting the exclude list.

But I'm not completely happy with this approach...

...since dynamically interpreting exclusion string from the user would be a pain, given the numerous exclusion formats supported by rsync... Personally I'm not sure I could implement a 100% affordable algorithm.

So what? (second take)?

Literally IDK. Every thought welcome. And if necessary I'm ready to re-explain, dig more, reword my confused pamphlet :P

[alessandro-fazzi]

Oh well... maybe a mv after the sync would also be the right trick for us?! 😅 I'm open to anything 🍡

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.