Bootstrapping users/groups

I’m currently having some trouble with users and groups: (running on 12.3.0)

#1. Adding users to a group fails
My goal is to extend the already existing user group ‘admin’ with a new user, for example ‘testuser1’. I’ve added the following lines to my main.yaml: (inside of the hcm-config)

definitions:
  config:
    /hippo:configuration/hippo:groups/admin:
      hipposys:members:
        operation: add
        type: string
        value: [someotheruser, testuser1]

[Documentation: https://www.onehippo.org/library/concepts/configuration-management/yaml-format.html]

When deploying to an environment with an already existing repository, nothing happens. While deploying with with -Drepo.bootstrap=true. Although, I could tell that the change is there, since the hcm:baseline did pickup the change, only the ‘admin’ group still does not have the new user added.

Any ideas on what is going wrong here?

#2. Why are users and groups not part of hcm-content?
When trying to add a new user group of a new user to the hcm-content, it will show you the following error:

Content node '/hippo:configuration/hippo:groups/test-group' in 'hippo-project/hippo-project/hippo-project-repository-data-application [content: configuration/groups/test-group.yaml]' is not categorized as content, skipping action 'APPEND'.

I don’t see why hippo:groups and hippo:users are not part of the content, they are editable by the users in the CMS. So why are they in the hcm-config?

#3. boostrap=full does not reload users and groups?

Since I had issues with the config of the user groups, I thought -Drepo.bootstrap=full might work.
[Documentation: https://www.onehippo.org/library/concepts/configuration-management/bootstrapping.html]
But this didn’t change anything for the hippo:groups, I thought, since they’re part of the hcm-config, they would also be recreated from the bootstrapped yaml’s. Isn’t this true? Or are there exceptions which are not documented?

.
I start thinking there might be something wrong with my repository, but it looks all fine…
Hope you guys can point me in the right direction! Looking forward to your replies!

Regards,
Jesper

2 Likes

For #2 You can explicitly mark them as content if you add the following lines in repository-data/application/src/main/resources/hcm-config/main.yaml :

#snip
    /hippo:configuration/hippo:groups:
      .meta:residual-child-node-category: content
    /hippo:configuration/hippo:users:
      .meta:residual-child-node-category: content
#snip

For #3: Are you deploying through rundeck? (ondemand1). There are deployment scripts that backup and recover users/groups. That might be an issue

Users/groups created in the cms are marked as “system”. The hippo:users/hippo:group nodes are marked as config. Content is not included in the default dist, but some users are required initially. However, users/groups generally should not be managed in the project code, that would be a security risk. As @bcanvural stated, on our own platform we backup users/groups/domains before a deploy and restore them after. This prevents malicious actors from injecting users through code.

@jasper.floor & @bcanvural Thanks for your replies.

To start with the simple question about #3; We’re not using rundeck, we have our own hosting. But I guess, labeling groups as content will prevent it from reloading, so then there is no need for updating. But let’s first tackle the problem of reloading at all.

My main problem is understanding the new YAML bootstrapping, I miss the overview I use to have with XML. If I read correctly, all changes in the hcm-config should be reloaded automatically when using -Drepo.bootstrap=[true|full]. But when we deploy for example a change in a groovy updater, we do not see any changes. Is there maybe something wrong with our repository or deployment in that case? I’m a bit lost here.

Also, when changes are picked up into the hcm:baseline (which I think is a kind of replacement for the hippo:initialize we use to have), why are they not always processed? Do we need to use -Drepo.bootstrap=full to reload the groovy updater for example?

For my #1 question; Is it possible to do a property modification (previously known as propset) multiple times?

with bootstrap=full, I would expect changes to be reloaded. In fact, everything should be reloaded. With true I would expect changes to be loaded. So if that isn’t happening then something it wrong. Only content nodes should not be reloaded. A full bootstrap should not generally be necessary.

For configuration, no propset is necessary because the definition belongs to the code. The whole definition is in the code and if the deployed artifact differs from the mainline it should be loaded.

For content, it is only loaded once ever. You can use content actions to reload content however. These are versioned. As you’ve discovered, content action only apply to nodes marked as content.

System nodes belong to the running instance. That should never be touched. If you created your groovy in the cms, then it is marked as system I believe. Otherwise I don’t know why it isn’t reloaded.

Thanks for the link, I didn’t know this.
Could it be that my groovy updaters changes don’t get recognized? Since it consist out of 2 resource files(1 json file for the parameters, 1 groovy file for the script) and a single yaml file to initiate the updater as a node into the queue. And that the changes are made in the json file, but the bootstrap mechanism only checks for changes in yaml?

-Drepo.bootstrap=full should practically never be needed, its mainly provided for specific scenarios in unit tests etc.
-Drepo.bootstrap=true will only apply differences between your current loaded yaml configuration and the previous stored mainline configuration, while -Drepo.bootstrap=full will simply (re)apply all the current loaded yaml configuration. The latter thus will also overwrite (replace) any runtime (config) changes made, not reflected in the mainline configuration.
For production you should never need and use -Drepo.bootstrap=full as you then will lose any runtime changes!

For the #1 question, your provided yaml configuration actually looks fine to me, so something else must go wrong here. Please double check any warnings logged as anything skipped/ignored typically does get logged.

With regards to multiple property modification operations: yes, you can/should use add (as you did), while the default is replace (which would be similar to the earlier xml propset).

Finally, concerning the groovy updater: can you show your yaml config to load the script + parameters?
And why did you define the parameters separately in json, not within the yaml file?
Also note that any children of the hippo:queue node will be treated as content, so change detection is not relevant (done) on them. But as the updater is supposed to process (and then empty) the queue, there shouldn’t be existing scripts in the queue at startup anyhow.

Actually, disregard my last comment about “children of the hippo:queue node will be treated as content, so change detection is not relevant (done) on them”. That is actually not correct and not even relevant here. I suspect something wrong with your yaml configuration for bootstrapping the updater though.

Also note that bootstrapping updater scripts in the queue should be done using hcm-content, not hcm-config, as they are intended to be loaded/bootstrapped only once!
If you did this before, updating that hcm-content script won’t reload (as hcm-content won’t get reloaded/updated). Instead you should add an action in hcm-actions.yaml to reload that content!

Hi Ate, thanks for your reply!

Ye exactly, that’s what I thought too, we always use -Drepo.bootstrap=true so that’s ok :slight_smile:

Error/warning wise we are only getting one ERROR → ERROR [org.onehippo.cm.engine.ConfigurationContentService.apply():174] Processing 'APPEND' action for content node '/content/documents/hippo-project/config/mail' failed.
This is because we changed something in our content, but it will not be processed. Personally it looks more like an warning to me, but maybe I see this wrong? If I want to change content locally, I should have done this in the development module, but still, nothing is breaking right?

My ResourceBundleUpdater.yaml:

definitions:
  config:
    /hippo:configuration/hippo:update/hippo:queue/ResourceBundleUpdater:
      jcr:primaryType: hipposys:updaterinfo
      hipposys:batchsize: 100
      hipposys:description: "The ResourceBundleUpdater can be used to \r\n1. Add new labels\
         \ to an EXISTING resource bundle\r\n2. Update existing labels from an EXISTING\
         \ resource bundle\r\n3. Remove labels from an EXSITING resource bundle\r\n\r\n\
         JSON Structure should be:\r\n{\"bundleId\": {\"resourceKey\": \"resourceValue\"\
         }}\r\n\r\nFor removing labels, set value to \"DELETE\""
      hipposys:dryrun: false
      hipposys:parameters:
        type: string
        resource: ResourceBundleUpdater.json
      hipposys:query: /jcr:root/content/documents/hippo-project/config/label//element(*,resourcebundle:resourcebundle)
      hipposys:script:
        type: string
        resource: ResourceBundleUpdater.groovy
      hipposys:throttle: 500
      hipposys:loglevel: DEBUG

This yaml loads the updater into the queue, so that work’s fine, only when making changes in the ResourceBundleUpdater.json, those are not processed.
The reason for using a json file is readability, since these values have to be changed quite often.

… I just see your other replies incoming :slight_smile: I didn’t know that we should move the updaters to the hcm-content module. Why should the updaters be part of the content? I think we, developers, should be in control of these scripts right? Can I mark them somehow, so they will be reloaded automatically? Maybe with something like .meta:category: config?

I’ve dived a bit deeper into this now.

Problem #1: the /hippo:groups/admin hipposys:members property is categorised as system, meaning value definitions will only be applied during initial bootstrap, but never updated.
See: https://www.onehippo.org/library/concepts/configuration-management/system-properties-with-initial-value.html
So while your yaml file looks OK, it actually is intentionally ignored (without warning in this case).
See the above linked documentation how you can still change this behavior, if you really must.

Problem #3: you’re indeed hitting upon another ‘silent’ skip/ignore, which IMO in this case should produce a warning (see below).
This is indeed the result of bootstrapping hcm config into the hippo:queue, and the reason why you should use hcm content for this purpose.
The hippo:queue is processed automatically, and emptied after startup, meaning the whole updater node definition will no longer exist thereafter.
With -Drepo.bootstrap=true, the logic of applying differences between baseline and current config is only (and only can be) executed when the ‘context’ of that change still exists in the repository.
In your case, your change is only on a property of the updater node, but that node itself already no longer exists.
Our logic and reasoning is that whenever something/someone removes specific configuration from the repository, that still takes precedence over (child) bootstrap configuration, e.g. we cannot and won’t ‘recreate’ the missing parent tree in that case, and thus skip/ignore such bootstrapped changes.
The ‘bug’ here IMO is that we currently don’t log a warning about this.
Side note: using -Drepo.bootstrap=full would/should recreate the updater node.

Anyhow, the above is why we explicitly describe on https://www.onehippo.org/library/concepts/update/updater-scripts.html that you should use hcm content instead, which indeed forces you to use a reload action, but then at least will reload the whole definition, not only try (and fail) to apply only a property change.

While using hcm-content here should fix your problem, the incorrect silent ignored/skipped hcm-config delta change in case of missing parent ‘context’ still is something we may want to ‘fix’ by logging a warning.
It would be great if you can create a JIRA ticket for this.

HTH, Ate

Hi everyone,

it’s an old thread, but we currently have a few issues with our JCR losing nodes or properties, probably related to bootstrapping. Thsu this thread was very helpful to better understand the process. What I would like to know, has the mentioned JIRA ticket ever been written and the log message been implemented? Would be great to know if we would recognize such cases directly in the logs or not.

Regards,
Jan

Hi Jan,

I don’t think the bug was ever created, this was considered as more of an edge case and not a serious issue. There is a small possibility that this could have been fixed already during some other refactoring and changes which may have happened since.

If you are facing any such issues we suggest you to file a ticket on our Support Portal with relevant details and we can investigate deeper into this.

Regards,
Richa

1 Like