Info [move content catalogs to reside in /library] #3052

jvonau · 2021-11-27T05:51:25Z

move the catalogs to reside in /library

holta · 2021-11-27T11:52:02Z

Interesting idea.

If there was ever an IIAB containing /library2 and /library3 (detachable drives possibly?) this could help make the lineage of all content (within each detachable drive?) more clear?

tim-moody · 2021-11-27T12:26:44Z

what problem does this solve?

jvonau · 2021-11-27T13:44:39Z

what problem does this solve?

No problem just better engineering

Interesting idea.

If there was ever an IIAB containing /library2 and /library3 (detachable drives possibly?) this could help make the lineage of all content (within each detachable drive?) more clear?

Well sort of, not so much concurrently using multiple /libraryX rather grouping of the info of what is contained under /library, used to be on a different partition just want to keep the metadata closer to the data. I propose that returning to the complete isolation of /library via a loop-backed filesystem proposed in #3051 we can gain easy portability of the 'data partition' within iiab with backing up one file being the main advantage. The data is architecture and OS agnostic should be able to export/import the data image file from one machine to another by just copying the 'data partition' and mounting the image file at /library. The advanced aspects of this design would allow creation of the 'data image' on one machine for deployment with multiple reuse on others, think of a finished preset ready to re-deploy on any machine with just one file to move into place. Using the image file method one can set a hard limit on how big /library can get.

tim-moody · 2021-11-27T13:57:45Z

I don't regard metadata that is intended to be consumed by a machine as in the same category as html that is intended to be consumed by a human, My understanding of the info directory is that it holds documentation in html for reading by a human.

In terms of restructuring, do you plan to move all of /etc/iiab to /library? For me ini, xml, and json are in similar categories.

jvonau · 2021-11-27T14:53:29Z

There is a movement upstream that would like to see /etc/ be devoid of most files and mounted read-only, the move of /etc/iiab -> /library/info would be in line with that thinking. Sure that is possible but would entail altering the 'iiab' curl script for the new location I didn't want to go that far in one shot but if that is a goal it is reachable.

tim-moody · 2021-11-27T14:59:26Z

There is a movement upstream that would like to see /etc/ be devoid of most files and mounted read-only

don't many apps have configuration files in /etc/, apache2, nginx, php, mysql, ssh, to mention a few? do they all have to move?

jvonau · 2021-11-27T15:10:33Z

Ah, light bulb moment users read docs served up at http://box/info stored at /library/www/html/info is what Tim is referring to while /library/info as I propose is a different physical path on the filesystem.

There is a movement upstream that would like to see /etc/ be devoid of most files and mounted read-only

don't many apps have configuration files in /etc/, apache2, nginx, php, mysql, ssh, to mention a few? do they all have to move?

What the future holds is fuzzy as the disros juggle things around, I don't see the files 'moving' soon but I can see /etc/ being read-only and files introduced using a bindmount routine like how snaps are put together.

holta · 2021-12-08T22:51:58Z

I don't fully understand the /library/iiab proposal just yet, so my apologies I can't comment intelligently.

But no matter what path/directory is chosen in the end (/library/etc or /library/catalogs or whatever makes most sense) to store these settings/catalog files — we should try to align the variable name (the variable that's currently named content_info) to correspond to that actual path.

As best we can anyway! Background below from: https://github.com/jvonau/iiab/blob/46d9191bacacd0e58a05fc637ebc09802b502674/vars/default_vars.yml#L36-L37 :

content_base: /library
content_info: "{{ content_base }}/iiab"

jvonau · 2021-12-08T22:59:23Z

I don't regard metadata that is intended to be consumed by a machine as in the same category as html that is intended to be consumed by a human, My understanding of the info directory is that it holds documentation in html for reading by a human.

Sure I can see the confusion, moved away from naming the directory "info" to using "iiab"

In terms of restructuring, do you plan to move all of /etc/iiab to /library? For me ini, xml, and json are in similar categories.

/etc/iiab -> /etc/local/iiab would make sense but that has to come with a change to the 'iiab" curl script, not required to have but should be on the to-do list.

Extracted from #3051 (comment)

jvonau@box:/media/usb1/output$ grep lrw maps-manifest.txt
lrwxrwxrwx 1 root     root           18 Dec  5 08:47 /library/www/html/common/assets/iiab.ini -> /etc/iiab/iiab.ini
lrwxrwxrwx 1 root     root           28 Dec  5 09:04 /library/www/html/common/assets/kiwix_catalog.json -> /etc/iiab/kiwix_catalog.json
lrwxrwxrwx 1 root     root           26 Dec  5 11:39 /library/www/html/common/assets/map-catalog.json -> /etc/iiab/map-catalog.json
lrwxrwxrwx 1 root     root           29 Dec  5 09:04 /library/www/html/common/assets/oer2go_catalog.json -> /etc/iiab/oer2go_catalog.json
lrwxrwxrwx 1 root     root           22 Dec  5 08:58 /library/www/html/common/assets/regions.json -> /etc/iiab/regions.json
lrwxrwxrwx 1 root     root           57 Dec  5 11:41 /library/www/osm-vector-maps/maplist/assets/bboxes.geojson -> /library/www/osm-vector-maps/viewer/assets/bboxes.geojson
lrwxrwxrwx 1 root     root           57 Dec  5 11:41 /library/www/osm-vector-maps/maplist/assets/countries.json -> /library/www/osm-vector-maps/viewer/assets/countries.json
lrwxrwxrwx 1 root     root           22 Dec  5 11:39 /library/www/osm-vector-maps/maplist/assets/regions.json -> /etc/iiab/regions.json
lrwxrwxrwx 1 root     root           26 Dec  5 11:39 /library/www/osm-vector-maps/test-page/assets/map-catalog.json -> /etc/iiab/map-catalog.json
lrwxrwxrwx 1 root     root           64 Dec  5 11:40 /library/www/osm-vector-maps/viewer/tiles/planet_z0-z6_2019.mbtiles -> /library/www/osm-vector-maps/installer/planet_z0-z6_2019.mbtiles

Would like to thank the effort put into soft-coding, made for juggling these 4 json file pretty straight forward. Think this PR and the companion one in iiab.-admin can be curl tested soon. I'll give that a spin and let you know.

tim-moody · 2021-12-08T23:52:11Z

/etc/iiab -> /etc/local/iiab would make sense

I take it this is on the model of /usr/local/bin, etc. will that be read only?

If there was ever an IIAB containing /library2 and /library3 (detachable drives possibly?) this could help make the lineage of all content (within each detachable drive?) more clear?

need to be careful about catalogs vs locally installed content, both are metadata but have different scope. library.xml records all zims installed, but that is different from the zim catalog downloaded from kiwix. btw, adm cons creates a local library.xml on any usb attached drive to allow copying of content on and off of the drive. for now there is no oer2go equivalent of library.xml, but it will probably be needed one day.

I'm still not clear why all of a sudden these catalogs need to be moved, but ini and other xml files don't.

tim-moody · 2021-12-08T23:56:56Z

Think this PR and the companion one in iiab.-admin can be curl tested soon. I'll give that a spin and let you know.

@holta please report what you tested along with the results. The impact goes substantially beyond ansible.

holta · 2021-12-09T00:14:37Z

@tim-moody wrote:

@holta please report what you tested along with the results. The impact goes substantially beyond ansible.

I haven't tested this PR to date.

(Did you mean to ask @jvonau?)

jvonau · 2021-12-09T01:10:01Z

/etc/iiab -> /etc/local/iiab would make sense

That should of been /usr/local/etc/iiab/, slight brain cramp with the quick reply.

I take it this is on the model of /usr/local/bin, etc. will that be read only?

No, but better adherence to https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard, note the /usr/local/

If there was ever an IIAB containing /library2 and /library3 (detachable drives possibly?) this could help make the lineage of all content (within each detachable drive?) more clear?

need to be careful about catalogs vs locally installed content, both are metadata but have different scope. library.xml records all zims installed, but that is different from the zim catalog downloaded from kiwix. btw, adm cons creates a local library.xml on any usb attached drive to allow copying of content on and off of the drive. for now there is no oer2go equivalent of library.xml, but it will probably be needed one day.

I'm still not clear why all of a sudden these catalogs need to be moved, but ini and other xml files don't.

Much for the same reason local_vars were suddenly renamed, before release. Now json is primarily used for transmitting data between a web application and a server, has much more to do with other data within /library and interfacing with the webserver than residing in /etc. Symlinks are provided within /etc/iiab to the new locations to cover some of the butt encase I missed something and keeps the docs the same.

And I'm giving this the combined curl test curl d.iiab.io/install.txt | sudo bash -s 3052 348 as I type this, I'll sum it up in tomorrow's meeting.

jvonau · 2021-12-09T04:42:38Z

Installed and running at 10.8.0.74, http://sprunge.us/9cOQRh
After selecting the 'test' preset I can see the jobs queue. I can see the modules selected/tagged as downloading but artofproblemsolving_en_all_maxi zim file is not being shown as ticked off. Waiting on the jobs.

jvonau · 2021-12-09T05:52:14Z

Noted that kalite was installed by the 'test' preset but not the en.zip just the es.zip, by design? all 14 jobs completed.

tim-moody · 2021-12-09T13:06:43Z

Noted that kalite was installed by the 'test' preset but not the en.zip just the es.zip, by design? all 14 jobs completed

That would be a bug. I assume this was not on an image that already had kalite. The EN lang pack is not installed if already present.

tim-moody · 2021-12-09T13:07:51Z

@holta

I haven't tested this PR to date.
(Did you mean to ask @jvonau?)

I got lost on the thread and thought you had volunteered to test.

jvonau · 2021-12-09T13:36:09Z

Noted that kalite was installed by the 'test' preset but not the en.zip just the es.zip, by design? all 14 jobs completed

That would be a bug. I assume this was not on an image that already had kalite. The EN lang pack is not installed if already present.

No bug, the install flags are present. IP moved now at 10.8.0.86

tim-moody · 2021-12-09T13:51:39Z

No bug

? si o no

I notice in the kalite admin that the en lang pack seems to be installed.

jvonau · 2021-12-09T14:02:36Z

I had to scroll down slightly to be able to see the drop-down language box near the bottom, both are present within kalite.

whitespace

holta · 2022-05-30T18:26:52Z

Whether or not this PR #3052 should be part of IIAB 8.0 I've relabeled it, so these design questions are at least more findable ✔️

correct - use {{ content_info }}

jvonau mentioned this pull request Nov 27, 2021

move metadata to info dir in /library iiab/iiab-admin-console#438

Open

holta changed the title ~~Info~~ Info [move content catalogs to reside in /library] Nov 27, 2021

holta added enhancement question labels Nov 27, 2021

holta added this to the 8.0 milestone Nov 27, 2021

jvonau and others added 5 commits December 8, 2021 16:32

info for metadata

12b051b

info for metadata2

2f8556a

info for metadata3

b3ee9e8

correct softcode hint

715091c

info -> iiab

46d9191

jvonau force-pushed the info branch from 0cbdc48 to 46d9191 Compare December 8, 2021 22:33

info -> iiab2

cf672bf

holta modified the milestones: 8.0, 8.1 Jan 1, 2022

jvonau mentioned this pull request May 30, 2022

How can I permanently attach an external USB drive, to add more content? [also: Admin Console preset drive out-of-space issues][also: Admin Console's GET-SPACE-AVAIL sometimes fails] #3106

Closed

Merge branch 'master' into info

d73fdc6

holta modified the milestones: 8.1, 8.0 May 30, 2022

Update fl.yml

291d59b

whitespace

Update install.yml

79c1c3b

correct - use {{ content_info }}

andrewclark5 mentioned this pull request Jun 9, 2022

Best strategy for preparing multiple IIAB servers using imaging #3247

Open

holta modified the milestones: 8.0, 8.1 Oct 23, 2022

holta modified the milestones: 8.1, 8.2 Jan 1, 2023

holta modified the milestones: 8.2, 8.3 Jan 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Info [move content catalogs to reside in /library] #3052

Info [move content catalogs to reside in /library] #3052

jvonau commented Nov 27, 2021

holta commented Nov 27, 2021

tim-moody commented Nov 27, 2021

jvonau commented Nov 27, 2021

tim-moody commented Nov 27, 2021

jvonau commented Nov 27, 2021

tim-moody commented Nov 27, 2021

jvonau commented Nov 27, 2021

holta commented Dec 8, 2021

jvonau commented Dec 8, 2021

tim-moody commented Dec 8, 2021

tim-moody commented Dec 8, 2021

holta commented Dec 9, 2021

jvonau commented Dec 9, 2021

jvonau commented Dec 9, 2021 •

edited

jvonau commented Dec 9, 2021

tim-moody commented Dec 9, 2021

tim-moody commented Dec 9, 2021

jvonau commented Dec 9, 2021

tim-moody commented Dec 9, 2021

jvonau commented Dec 9, 2021

holta commented May 30, 2022

Info [move content catalogs to reside in /library] #3052

Are you sure you want to change the base?

Info [move content catalogs to reside in /library] #3052

Conversation

jvonau commented Nov 27, 2021

holta commented Nov 27, 2021

tim-moody commented Nov 27, 2021

jvonau commented Nov 27, 2021

tim-moody commented Nov 27, 2021

jvonau commented Nov 27, 2021

tim-moody commented Nov 27, 2021

jvonau commented Nov 27, 2021

holta commented Dec 8, 2021

jvonau commented Dec 8, 2021

tim-moody commented Dec 8, 2021

tim-moody commented Dec 8, 2021

holta commented Dec 9, 2021

jvonau commented Dec 9, 2021

jvonau commented Dec 9, 2021 • edited

jvonau commented Dec 9, 2021

tim-moody commented Dec 9, 2021

tim-moody commented Dec 9, 2021

jvonau commented Dec 9, 2021

tim-moody commented Dec 9, 2021

jvonau commented Dec 9, 2021

holta commented May 30, 2022

jvonau commented Dec 9, 2021 •

edited