feature requests about ansible-truenas

[arensb]

The good news is that periodic snapshots are already there:
https://arensb.github.io/truenas/pool_snapshot_task_module.html#ansible-collections-arensb-truenas-pool-snapshot-task-module

I'll see if I can add SMART and scrub stuff, but I don't know when I'll be able to get to this.

In the meantime, maybe you can help me out by thinking about what these modules should look like: how would you like to write an Ansible playbook for them? What should the parameters be called, so as to make sense to the people using them, and also fit into the Ansible ecosystem. Are there any results that it would make sense to return, besides "it succeeded" or "it failed"?

[paulo-erichsen]

If I could configure those, I'd probably do something like this

As far as naming variables, I'd probably try to stay as close as possible to what the Web UI shows.

• https://www.truenas.com/docs/core/uireference/services/smartscreen/
• https://www.truenas.com/docs/core/uireference/tasks/smarttests/
• https://www.truenas.com/docs/core/uireference/tasks/scrubtasks/

---
- name: Configure SMART disk temperature checks
  arensb.truenas.smart:
    check_interval: 30
    power_mode: never
    difference: 0
    informational_temp: 0
    critical_temp: 45

# I'm not sure if the following 2 should be merged into a single array or kept separately
- name: Add SMART short disk periodic tests
  arensb.truenas.pool_smart_task:
    disks: all
    type: short
    description: "run SHORT tests on the 7th, 17th and 27th of each month at 1am"
    schedule: "0 1 7,17,27 * *"

- name: Add SMART long disk periodic tests
  arensb.truenas.pool_smart_task:
    disks: da0,da1,da2,da3
    type: long
    description: "run LONG tests on the 1st and 15th of each month at 1am"
    schedule: "0 1 1,15 * *"

# similarly for this one: there can be multiple pools that need scrubs, need to figure out if arrays is what we want
- name: Add monthly scrub tasks to the tank pool
  arensb.truenas.pool_scrub_task:
    enabled: true
    pool: tank
    threshold_days: 22
    description: "run Scrub tasks on the 19th of each month at 1am"
    schedule: "0 1 19 * *"

If those tasks were to allow arrays, here's what I think they could look like

# Here's what it would look like if these allowed arrays
- name: Add SMART disk periodic tests
  arensb.truenas.pool_smart_task:
    - disks: all
      type: short
      description: "run SHORT tests on the 7th, 17th and 27th of each month at 1am"
      schedule: "0 1 7,17,27 * *"
    - disks: da0,da1,da2,da3
      type: long
      description: "run LONG tests on the 1st and 15th of each month at 1am"
      schedule: "0 1 1,15 * *"

lastly, we might need to add state: absent or state: present to some of these scheduled tasks

[arensb]

Thanks. One consideration here is: how can we tell whether a job on the TrueNAS system is the same one as the one defined in an Ansible play? That is, if the play says to run a short SMART test on disk da0 every Monday, and the TrueNAS system currently has one job: a short SMART test on disk da0 every Tuesday, does that mean that the existing job should be changed to run on Mondays instead of Tuesdays? Or that we should add a second job, one that runs on Mondays, and ignore the Tuesday job?

I'm pretty sure I had to solve this same problem for NFS mounts or something, but I'm open to other people's ideas.

[arensb]

I've added a smart module on branch feature/pr7-smart. Usage docstrings are at the top of the module.

Can you please try it out and see if it works for you?

[paulo-erichsen]

One consideration here is: how can we tell whether a job on the TrueNAS system is the same one as the one defined in an Ansible play?

The way I see it, there's a couple of ways to go about this.

approach 1

I wouldn't mind if the module required users to add the object id. The first time users run, if id is undefined, then CREATE, but if id is defined, then UPDATE.
If this approach was taken, it would be straightforward, but we'd need really good documentation about it, including how to find out the ID such that users can edit their playbooks accordingly.

Example:

- name: Add SMART disk periodic tests
  arensb.truenas.pool_smart_task:
    - id: null # <<<<< this will CREATE the object since we haven't defined which object to change.
               # Prior to a subsequent run, the user is expected to update the id such that TrueNAS doesn't create duplicates
      disks: all
      type: short
      description: "run SHORT tests on the 7th, 17th and 27th of each month at 1am"
      schedule: "0 1 7,17,27 * *"
    - id: 2  # <<<< this will update object with = 2, or return an error if it doesn't exist
      disks: da0,da1,da2,da3
      type: long
      description: "run LONG tests on the 1st and 15th of each month at 1am"
      schedule: "0 1 1,15 * *"

approach 2

Another approach without specifying ids would be for the module itself to try to create an "primary key" to find the right object. Example: maybe pool_smart_task could use a primary key of "description" (similar to what was done to the snapshots module). Or maybe the pool_scrub_task could have a primary key of "pool+description"

IMO approach 1 is the more straightforward - it may be more difficult for users since they'd need to specify the ids in their playbooks, but this has the least amount of gotchas or surprising behaviors

[paulo-erichsen]

I've added a smart module on branch feature/pr7-smart. Usage docstrings are at the top of the module.

Can you please try it out and see if it works for you?

I'll try it out and let you know :)

EDIT

Ok, I've tested it a bit and it's able to set all values, except for "interval". It currently seems to ignore the "interval" setting that was passed in.

This is how I tested:

- name: Configure SMART disk temperature checks
  arensb.truenas.smart:
    interval: "50"
    power_mode: "idle"
    temp_difference: "51"
    temp_info: "52"
    temp_crit: "53"

[paulo-erichsen]

Back to my first comment, when choosing between supporting arrays or not supporting arrays, I think it would be simpler if NO arrays were supported.

Looking into some examples how official ansible modules are done, I see this

• https://docs.ansible.com/ansible/latest/collections/ansible/builtin/user_module.html

- name: Add the user 'johnd' with a specific uid and a primary group of 'admin'
  ansible.builtin.user:
    name: johnd
    comment: John Doe
    uid: 1040
    group: admin

- name: Add the user 'james' with a bash shell, appending the group 'admins' and 'developers' to the user's groups
  ansible.builtin.user:
    name: james
    shell: /bin/bash
    groups: admins,developers
    append: yes

Notice how each user is modified separately instead of through a single call

[arensb]

I wouldn't mind if the module required users to add the object id.

From what I've seen, the TrueNAS middleware assigns IDs for you, and doesn't allow you to choose them. So that's not an option.

But it does look similar to something I had to do elsewhere, in the sharing_nfs module, I think: there, you have to specify the path to export, and the export flags. So if the flags on the NAS don't match the ones in the Ansible play, what should the module do?

The way I solved it there was to require a description (similar to the "Description" field in S.M.A.R.T. test tasks). For state == present, it would look for an export with a matching description.

Also, for scheduled tasks, I'd like to stick to the same parameters as builtin.cron, for consistency.

[paulo-erichsen]

From what I've seen, the TrueNAS middleware assigns IDs for you, and doesn't allow you to choose them. So that's not an option.

right. What I was thinking was when the user wanted to update an existing the object, the user would need to manually edit the playbook to refer to the ID stored in the database. Example, the user would need to run midclt call pool.scrub.query | jq, take note of the IDs and then write the playbook using the IDs. Also when the user created the item through the playbook, the module should return the new ID.

But I understand that this approach can be complicated for users since they would need to know how to lookup the IDs and also modify their playbooks to differentiate between CREATE and UPDATE.

Either way, @arensb what you're doing is great! I could see it going either way. I'd suggest that once you add these and get a new release out, to definitely share them on reddit and discord. (I've found your post on reddit and that's how I ended up here)

[arensb]

I've added a new branch, feature/pr7-smart-test-task with a draft smart_test_task module. Give it a shot and see if it works.

You'll have to figure out the playbook from the docstring at the top. I haven't finished the examples and output sections.

[paulo-erichsen]

I've added a new branch, feature/pr7-smart-test-task with a draft smart_test_task module. Give it a shot and see if it works.

Nice! I'll check it out for sure, but it will be a few days until I get to it since I'll be out of town

[paulo-erichsen]

done some testing

feature/pr7-smart

@arensb I don't know if you had a chance to see this comment, but this was one issue I had found

Ok, I've tested it a bit and it's able to set all values, except for "interval". It currently seems to ignore the "interval" setting that was passed in.

feature/pr7-smart-test-task

it seems this also has another issue. My config is this

- name: Add SMART short disk periodic tests
  arensb.truenas.smart_test_task:
    name: "run SHORT tests on the 7th, 17th and 27th of each month at 1am"
    disks: ALL
    test: short
    minute: 0
    hour: 1
    day: "7,17,27"
    month: "*"
    weekday: "*"

- name: Add SMART long disk periodic tests
  arensb.truenas.smart_test_task:
    name: "run LONG tests on the 1st and 15th of each month at 1am"
    disks: ada0,ada1
    test: long
    minute: 0
    hour: 1
    day: "1,15"
    month: "*"
    weekday: "*"

but when I run it returns this

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'schedule'
fatal: [[email protected]]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 192.168.122.100 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/root/.ansible/tmp/ansible-tmp-1693974349.8145535-3600-172060455326188/AnsiballZ_smart_test_task.py\", line 107, in <module>\r\n    _ansiballz_main()\r\n  File \"/root/.ansible/tmp/ansible-tmp-1693974349.8145535-3600-172060455326188/AnsiballZ_smart_test_task.py\", line 99, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/root/.ansible/tmp/ansible-tmp-1693974349.8145535-3600-172060455326188/AnsiballZ_smart_test_task.py\", line 47, in invoke_module\r\n    runpy.run_module(mod_name='ansible_collections.arensb.truenas.plugins.modules.smart_test_task', init_globals=dict(_module_fqn='ansible_collections.arensb.truenas.plugins.modules.smart_test_task', _modlib_path=modlib_path),\r\n  File \"/usr/local/lib/python3.9/runpy.py\", line 225, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File \"/usr/local/lib/python3.9/runpy.py\", line 97, in _run_module_code\r\n    _run_code(code, mod_globals, init_globals,\r\n  File \"/usr/local/lib/python3.9/runpy.py\", line 87, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/tmp/ansible_arensb.truenas.smart_test_task_payload_r380z7wa/ansible_arensb.truenas.smart_test_task_payload.zip/ansible_collections/arensb/truenas/plugins/modules/smart_test_task.py\", line 382, in <module>\r\n  File \"/tmp/ansible_arensb.truenas.smart_test_task_payload_r380z7wa/ansible_arensb.truenas.smart_test_task_payload.zip/ansible_collections/arensb/truenas/plugins/modules/smart_test_task.py\", line 237, in main\r\nKeyError: 'schedule'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

this was fixed with adding this line

# Collect arguments to pass to resource.create()
arg = {
    'desc': name,
+    'schedule': {},
}

feedback

IMO the ID that ansible uses in smart_test_task.py may not be best to use description since the UI doesn't even require that parameter. I think it would be better if it was possible to make the ID based on DISKS+TYPE, but that has it's own cons

# currently this is used
smart_test_info = mw.call("smart.test.query",
                          [["desc", "=", name]])

# maybe could be something like this - the challenge with the following is that the disks might not be ordered the same? (I haven't checked if the order is always the same)
$ midclt call smart.test.query '[["type", "=", "LONG"], ["disks", "=", ["{serial}555666777", "{serial}111222333"]]]'

But there's pros/cons to each approach, for example

• id: description --> simpler, but when a user changes the description and runs the playbook, a new entry is created
• id: disks+type --> more complex, but when a user adds more disks and runs the playbook, a new entry is created

I suppose using the description as an id should be fine as long as documentation is top notch

feature/pr7-scrub-task

looks good ✔️

[arensb]

It currently seems to ignore the "interval" setting that was passed in.

D'oh! This is fixed now, on branch feature/pr7-smart.

[arensb]

I was able to reproduce the bug in feature/pr7-smart-test-task, and fix it. As you suspected, it had to do with a messed-up schedule field. (Or, more precisely, I found a bug in the schedule-processing code while editing the "if the resource exists" section, but then didn't apply the fix correctly to the "if we need to create the resource" section.) Please give it another try.

IMO the ID that ansible uses in smart_test_task.py may not be best to use description since the UI doesn't even require that parameter. I think it would be better if it was possible to make the ID based on DISKS+TYPE, but that has it's own cons

It looks as though the required fields to create a S.M.A.R.T. Test task are the test type ("LONG", "SHORT", etc.) and either all_disks or a list of disks. However, that does not uniquely identify a job: you can have several LONG jobs on disk ada0.

In fact, there may be valid reasons for doing this: maybe you have several TrueNAS machines, and you want to run a SHORT test on ada0,ada1 on all machines on the 1st of each month. But some of them are backup servers that keep user data on those disks, and you want to test them on the 15th of each month as well:

- hosts: truenas-hosts
  tasks:
    - arensb.truenas.smart_test_task:
        disks: [ ada0, ada1 ]
        test: short
        day: "1"
- hosts: backup-servers
  tasks:
    - arensb.truenas.smart_test_task:
        disks: [ ada0, ada1 ]
        test: short
        day: "15"

Assuming that backup-servers is a subset of truenas-hosts, we now have some hosts with two jobs with the same disks and type.

The purpose of an identifier is to be able to point to something (a task, in this case) and say "there: that's the one I'm talking about". That seems easier to do with the description field than with any other field or combination of fields. Yes, it means that you can't change the description without deleting and recreating the task, but hopefully that won't happen too often.

A bigger problem, to my mind, is that this module doesn't allow you to delete all tasks and reset the machine to a known state where the only tasks are the one defined in your playbook.

[paulo-erichsen]

That seems easier to do with the description field than with any other field or combination of fields. Yes, it means that you can't change the description without deleting and recreating the task, but hopefully that won't happen too often.

Agreed. that seems the most simple and straightforward way to handle this

[I fixed it,] please give it another try.

will do :)

DONE - looks good now

[arensb]

DONE - looks good now

Thanks. I think that just leaves the S.M.A.R.T. service itself. Have you had a chance to try that out?

[paulo-erichsen]

I did. The updated change that fixes the interval works! ✔️

[paulo-erichsen]

while I have your attention @arensb, I've opened another much smaller request #8

[paulo-erichsen]

closed with v1.8.0