fossrit / infrastructure

Set of scripts, Ansible playbooks/roles, and other tools to automate and manage FOSS@MAGIC infrastructure

FOSS@MAGIC infrastructure

Set of scripts, playbooks, and other tools to automate and manage FOSS@MAGIC infrastructure

About

This repository contains various scripts and other automation tools to manage the FOSS@MAGIC infrastructure. Changes to the infrastructure should be made via this repository.

How to use

As of now, there is only one playbook to use:

ansible-playbook-3 -u $USER -K playbooks/irc-lug.yml

Legal

Project license: BSD 3-Clause License

This repository is licensed under the BSD 3-Clause License. Reuse anything you want in your own configurations. If you publish any remixed work online, add a link back here to your README please. 📝 And you can say 👋 to me in your git commit. Tag me! @jwflory

Attributions

These projects are used, or were used at one time, by this project. Thanks for the work of those who shared changes in the open. 🎉

• ansible/ansible-examples (wordpress-nginx_rhel7)
• jwflory/ansible-role-teleirc

infrastructure's Issues

Better management for secrets

Per #13 and helpful feedback from @Qalthos:

You could use an inline vault string with either a vault password provided at runtime or a vault password file.

Investigate automatic deployment policy from master branch

Summary

Set up automatic Ansible playbook runs when new changes are pushed to this repo's master branch

Background

Currently, changes are still pushed manually by FOSSRIT sysadmins. You have to manually run a playbook when a change is made. It's more convenient and less work to maintain if there were a triggered event to deploy new changes when a pull request is merged to the master branch of this repo.

This way, anyone can contribute without having SSH access to the servers, and still see their changes go through once the PR is merged. The emphasis changes from SSH access to commit access on this repository.

Additional protections are required on master branch once this is set up.

Details

I see Fedora Infrastructure does this with a special machine dedicated only to running Ansible playbooks. I envision this step-by-step process for how it might work:

1. Pull request is merged into master on FOSSRIT/infrastructure
2. Webhook / trigger is sent to remote server (or cronjobs could be used?)
3. Ansible playbook run occurs on playbook server
4. Changes push out across infrastructure
5. Logs are captured in case of failure with Ansible playbook

Outcome

1. Moving responsibility to git commit access from remote account / SSH account access
2. Automated deployment process after a PR is merged that does not require humans (and outlasts my time as a student at RIT)

Service disruption: Slack bridge gone one-way

Summary

Slack messages are not currently going over to IRC side, but I can see IRC messages.
@jwflory was saying this is possibly due to his slack account privileges being downgraded, and since the slack bot is based off his account, it does not have the correct privileges any more.

Expected results

IRC <--> Slack

1. Send message on IRC, shows up on slack
2. Send message on slack, shows up on IRC

Actual results

Messages from slack do not show up in IRC chat

1. Send message on IRC, it shows up in slack
2. Send message on slack, it does not show up in IRC

Priority requested

• Urgency: high

Change repository name: tools => toolbox

I think it would make sense to rename this repository from "tools" to "toolbox". With PR #1, it expands the scope of this repository a little bit beyond just scripts and makes it more of a "grab-bag" of different tools that might be useful to current and future students. The name sounds a little more generic and applicable to about anything this way. 😄

I could change the name myself, but thought it would be better to ask the repository creator, @decause, before changing it.

Bridge `#rit` from IRC to Telegram

re: IRC discussion

Add templates for different types of issues

Adding different kind of issue templates for different needs would be helpful. Some examples:

1. Service disruption
2. Infrastructure request
3. Enhancement / improvement
4. Other (free-form)

Add continuous integration checks on YAML syntax in new PRs

Remove prefix on Slack bridge

The name of the Slack bridge bot (slack-ritlug) tells us where it's coming from, so the extra [slack] prefix is redundant

Slack bridge gone to one-way

Slack bridge works IRC -> Slack, but messages from the Slack side aren't going to IRC

Service disruption: Slack bridge down

Summary

bridge to slack has stopped working. No messages going either way
Expected results

Messages bring bridges both ways on relevant channels
Actual results

messages not being sent to either side, unknown if bot errored or died
Priority requested

Urgency: high/medium
Requested deadline: reasonably soon

Other details

Can we get auto-restarts, monitoring, or something for the bot?

Migrate Discourse from bare-metal to cloud hosting

Summary

Migrate existing Discourse installation to a cloud provider like Google Cloud / Amazon Web Services

Background

Currently the application is running on a bare metal server with more resources than it needs. Running Discourse in the cloud helps the site scale better as it grows and also is more economical purchase than our current fixed pool of resources.

Details

1. Backup existing data (easy to do in Discourse web interface)
2. Set up new cloud instance in cloud platform (I'm leaning towards Google)
3. Install Discourse in cloud environment
4. Import database / config dumps into new install
5. Change DNS records, leave old site available for ~1 week
6. Destroy Discourse site on current server once data and performance are guaranteed

Outcome

• Easier to scale / get new resources if the site grows
• More economical than renting a VPS with fixed pool of resources

Move #rit-foss back to latest TeleIRC release once new release is tagged

#rit-foss is currently being used as another testing bed for TeleIRC, running off HEAD for the remainder of these summer days. However, since this channel is used for classroom communications, we should move it back to the latest stable {{ default_version }} once the next release is out.

Service disruption: Matrix.org security breach (advisory, RESOLVED)

The matrix.org bridges to freenode, and thus to our IRC channels there (#rit-foss et al) was down

Expected results

Posts made on either side (withiin Matrix clients like Riot, or within IRC clients) are bridged to the other.

Actual results

No traffic was flowing while the matrix.org instance was down.

Priority requested

• Urgency: [low]

This has been resolved on the matrix.org side

https://matrix.org/blog/2019/04/11/security-incident/

Other details

This issue is mostly advisory, to explain lack of this specific connectivity in the recent past. Those who rely on matrix for some or all of their IRC presence may continue to be affected as they change passwords as recommended by the Matrix folk and reconnect clients.

Service disruption: Attempts to reply to Discourse by email bounce

Summary

Email replies to Discourse bounce

Expected results

!. Compose email reply to a Discourse topic/thread/whatever-they-are
2. Send
3. Reply is accepted and included in the discussion

Actual results

1. Compose, as above
2. Send, as above
3. Receive SMTP-time bounce message from one's own SMTP service

Priority requested

• Urgency: [high]
• Requested deadline: 2019-04-22

Other details

This appears to be what the Discourse SMTP server is sending my mail server:

Content-Description: Notification                                                                                                   
Content-Type: text/plain; charset=us-ascii                                                                                          
                                                                                                                                    
This is the mail system at host eforward3a.registrar-servers.com.                                                                   
                                                                                                                                    
I'm sorry to have to inform you that your message could not                                                                         
be delivered to one or more recipients. It's attached below.                                                                        
                                                                                                                                    
For further assistance, please send mail to postmaster.                                                                             
                                                                                                                                    
If you do so, please include this problem report. You can                                                                           
delete your own text from the attached returned message.                                                                            
                                                                                                                                    
                   The mail system                                                                                                  
                                                                                                                                    
<[email protected]>: unknown user:                                                         
    "[email protected]" 

So, it's clear enough to infer that the bit between the plus symbol and the at symbol is unique to the specific message to which one is trying to reply, and that there would not be a user with the entirety of that local part in any lookup table. Everything to the left of the @ is meant to be handled only by the local mail system. Plus addressing, as I understand it, is meant to be able to accomodate this sort of thing, but it may require some tweaking to the receiving MTA configuration to allow it to pass on through to discourse.

Change command prefix on fanshawe

Summary

Change the prefix that fanshawe uses for commands to something other than .

Background

Fanshawe sees any message that starts with . as a command and then tries to parse it. However, messages are often started with ..., which fanshawe will attempt to parse and then drop an error message in chat, which is annoying.

Details

Configuration change

Outcome

See Summary

Service disruption: Matterbridge went down with odd error

Summary

I had to restart the slack bridge this afternoon. Matterbridge went offline this morning with an odd error:

Sep 09 22:41:04 ritlug-irc matterbridge[7829]: time="2019-09-09T22:41:04-04:00" level=info msg="irc.freenode: joining #rit (ID: #ritirc.freenode)" prefix=irc
Sep 09 22:41:05 ritlug-irc matterbridge[7829]: time="2019-09-09T22:41:05-04:00" level=info msg="irc.freenode: joining #rit-foss (ID: #rit-fossirc.freenode)" prefix=irc
Sep 09 22:41:06 ritlug-irc matterbridge[7829]: time="2019-09-09T22:41:06-04:00" level=info msg="irc.freenode: joining #rit-lug-projects (ID: #rit-lug-projectsirc.freenode)" prefix=irc
Sep 09 22:41:08 ritlug-irc matterbridge[7829]: time="2019-09-09T22:41:08-04:00" level=info msg="irc.freenode: joining #rit-lug-sysadmin (ID: #rit-lug-sysadminirc.freenode)" prefix=irc
Sep 09 23:08:12 ritlug-irc matterbridge[7829]: time="2019-09-09T23:08:12-04:00" level=error msg="Connection failed "slack rate limit exceeded, retry after 59s" &slack.RateLimitedError{RetryAfter:59000000000}" prefix=slack
Sep 09 23:08:12 ritlug-irc matterbridge[7829]: time="2019-09-09T23:08:12-04:00" level=error msg="Connection failed "slack rate limit exceeded, retry after 59s" &slack.RateLimitedError{RetryAfter:59000000000}" prefix=slack
Sep 09 23:26:55 ritlug-irc matterbridge[7829]: time="2019-09-09T23:26:55-04:00" level=error msg="Could not retrieve bot information: &errors.errorString{s:"bot_not_found"}" prefix=slack
Sep 09 23:26:55 ritlug-irc matterbridge[7829]: time="2019-09-09T23:26:55-04:00" level=error msg="&errors.errorString{s:"bot_not_found"}" prefix=slack
Sep 10 07:43:01 ritlug-irc matterbridge[7829]: time="2019-09-10T07:43:01-04:00" level=error msg="Connection failed "slack rate limit exceeded, retry after 59s" &slack.RateLimitedError{RetryAfter:59000000000}" prefix=slack
Sep 10 07:43:01 ritlug-irc matterbridge[7829]: time="2019-09-10T07:43:01-04:00" level=error msg="Connection failed "slack rate limit exceeded, retry after 59s" &slack.RateLimitedError{RetryAfter:59000000000}" prefix=slack

Expected results

1. Matterbridge is online
2. I send a message on Slack
3. I see it on IRC

Actual results

1. Matterbridge is offline
2. I send a message on Slack
3. I don't see the message on IRC

Priority requested

• Urgency: medium
• Requested deadline: None

Create a "Student Projects" category on Discourse

Summary

Create a "Student Projects" category on Discourse for students to share personal projects with other students, faculty, and alums

Background

It would be a good place to consolidate activity about project work. It also encourages people to share more about their personal projects by making a specific space for that topic.

Details

A sub-category of RIT Academic Life I think?

Outcome

1. More people learn and discover on-going student project work
2. We can solicit more people to help us with Imagine RIT 🎉

Create a script for updating EoL XO laptops

In the last class of HFOSS today, we identified an issue that some XO laptops are having. Some of them, when flashed from stock, are using EoL repositories for Fedora 18. Because of that, it is impossible to update them or install other packages, like git or pygame.

A script could solve this problem by pulling down the correct repo files, the GPG key, importing it, and then updating the system. This shouldn't be too hard to do, and I will try to add it in soon.

CCing @ritjoe on this one.

email bounces

Just a head's up. The bounces indicate the messages were delivered and that these are, so far, only warnings, but given all the shenanigans I've heard regarding other email-related integrations I figured it was worth a mention.

These are all in response to @Nolski's exhortative message:

https://lists.fedoraproject.org/archives/list/[email protected]/message/OERPVSM427YUP5WLC3KDSQEILR56TTP2/

From my mutt inbox:

  26     Dec 03 admin@fedorapro ( 180) Uncaught bounce notification                                                                                                                                         
  27     Dec 03 admin@fedorapro ( 180) Uncaught bounce notification                                                                                                                                         
  28     Dec 03 admin@fedorapro ( 180) Uncaught bounce notification                                                                                                                                         
  29     Dec 03 admin@fedorapro ( 180) Uncaught bounce notification 

All four bounce notifications are for gmail users. I've redacted the local part of the email address here out of deference to the subscriber's privacy but this is representative of the bounces:

[-- Attachment #2 --]                                                                                                                                                                                       
[-- Type: message/rfc822, Encoding: 7bit, Size: 9.4K --]                                                                                                                                                    
                                                                                                                                                                                                            
Date: 03 Dec 2018 10:19:33 -0500                                                                                                                                                                            
From: RIT Postmaster <[email protected]>                                                                                                                                                                   
To: [email protected]                                                                                                                                                                  
Subject: Delivery Status Notification (Delay)                                                                                                                                                               
                                                                                                                                                                                                            
[-- Attachment #1 --]                                                                                                                                                                                       
[-- Type: text/plain, Encoding: quoted-printable, Size: 0.6K --]                                                                                                                                            
                                                                                                                                                                                                            
This is an automatically generated Delivery Status Notification.                                                                                                                                            
                                                                                                                                                                                                            
THIS IS A WARNING MESSAGE ONLY.                                                                                                                                                                             
                                                                                                                                                                                                            
YOU DO NOT NEED TO RESEND YOUR MESSAGE.                                                                                                                                                                     
                                                                                                                                                                                                            
Delivery to the following recipients has been delayed.                                                                                                                                                      
                                                                                                                                                                                                            
              <[email protected]>                                                                                                                                                                       
                                                                                                                                                                                                            
The reason for the problem:                                                                                                                                                                                 
4.3.2 - Not accepting messages at this time 421-'4.7.0 This message does not have authentication information or fails to pass\n4.7.0 authentication checks. To best protect our users from spam, the\n4.7.0 
message has been blocked. Please visit\n4.7.0  https://support.google.com/mail/answer/81126#authentication for more\n4.7.0 information. n189si6042746qkc.170 - gsmtp'                                       
                                                                                                                                                                                                            
                                                                             

Set up meaningful labels

Add ritlug-root to the GitHub org

ritlug-root is used for GitLab (inc. CI) on the RITlug side. I'd like to test deploying the fossrit website via GitLab Pages. To do this I need ritlug-root added to the org

[ansible] Define authorized users in global variables

Summary

Add list of authorized usernames / people to global variables, update any roles using target_user varariable

Background

This came up in #26. Originally, I set things up so a single target user is available in roles where privileges need to be granted to a user on the system. My original idea was that you could manually edit the variable if you needed a new target user, but this has two flaws:

1. Change is not documented or logged (no way to make note of who has access and who doesn't)
2. Current implementation of target_user is flawed anyways (didn't work well with SSHD role when we have multiple public keys for different users)

Details

A few things need to be done:

• MIGRATE list of users out of SSHD role into global group_vars (introduced in #26 as short-term fix)
• DELETE old vars: target_user and user_home_dir
• Update roles that utilize the old variables (grep -rnw . -e 'var_name')

Outcome

1. Better documentation of who has access to FOSSRIT infrastructure
2. Easier to maintain, update, and add new authorized users

Fix Telegraf on irc-lug

The telegraf monitoring service is currently installed and running on the irc-lug server, but is failing to run due to misconfiguration.

This should either be removed, or set up correctly so we have effective monitoring.

Write an Ansible playbook for Discourse

Summary

Create an Ansible playbook for DIscourse

Background

We want to make discourse easy to maintain, so we want to automate this

Details

Write a playbook in Ansible to control the deployment of Discourse and the configuration of it, then test in a vagrant VM before moving onto production deployment.

Outcome

We have a discourse site to use for FOSS classes that is easy to deploy and maintain

Service disruption: Slack bot died

Summary

bridge to slack has stopped working. No messages going either way

Expected results

Messages bring bridges both ways on relevant channels

Actual results

messages not being sent to either side, unknown if bot errored or died

Priority requested

• Urgency: high/medium
• Requested deadline: reasonably soon

Other details

Can we get auto-restarts, monitoring, or something for the bot?

Service disruption: discourse emails

Summary

Discourse emails are getting caught in the spam filters again

Expected results

Emails should not get flagged as spam

Actual Results

RIT's anti-spam gateway has determined this message may be spam results

Priority requested

• Urgency: high
• Requested deadline: soon

Other details

This may possibly be related to switching to the em#### subdomain for sending emails. I haven't checked the dns policy for emails, but I wrote a blog post on how to do so:

https://ctmartin.me/blog/2020/07/email-security/

github.com/ritjoe deprecated

Please be advised that https://github.com/FOSSRIT/hfoss exists. I have been inconsistent with merging ritjoe/hfoss back into the master branch of this repository but have done so at least once, 3 years ago.

More recently, I created the branch https://github.com/FOSSRIT/hfoss/tree/2015F-2020S as being up to date with what I had in ritjoe/hfoss at the end of the Spring 2020 semester (RIT term 2195 as per https://www.rit.edu/calendar/1920). I know of no reason this couldn't be merged into the FOSSRIT/hfoss master branch but I figured this was a cleaner, more obvious waypoint.

The third branch in FOSSRIT/hfoss (develop) I'll address in a different issue.

I saw some links in a document associated with the instance of HFOSS that started today (Spring 2021, RIT term 2205) pointing back to ritjoe/hfoss that would probably be better directed to one of the branches of FOSSRIT/hfoss

@Nolski @jwflory @whenbellstoll @itprofjacobs

readthedocs usage

FOSSRIT has two associated Read The Docs entities from before we began using @ryansb's ofcourse for HFOSS

The first is built from source held in @ralphbean's repository. Ralph and I both hold accounts with administrative access to this.

The second is built from source held in the develop branch of the FOSSRIT/hfoss repository Ralph, Remy (@decause), and I hold accounts with administrative access to this.

I mentioned this latter, FOSSRIT/hfoss, branch briefly in #87

I mention this a) so that people know it's there b) so that no one inadvertently makes breaking changes to it through the HFOSS/hfoss repo and c) so that we can make changes to its administration as are seen fit to do.

Bridge RITlug #projects to freenode

Currently, most RITlug slack channels are available on freenode. One withstanding channel is #projects. This would enable even further contributions from those on either side of the bridge.

This is related to this RITlug Infrastructure issue.

Consider project license change to BSD 3-Clause License

Summary

Change project license from Mozilla Public License 2.0 to BSD 3-Clause License

Background

This project is licensed under the Mozilla Public License 2.0 (MPLv2), a copyleft license. The MPLv2 requires changes made to upstream software to also be released open source. This license promotes the practice of participating in an upstream project.

However, given this repo contains configurable assets that must be changed to be useful (e.g. variables and config files), the MPLv2 is a restrictive license for reuse. Some may steer away because the license implies they must release their automation infrastructure tools as open source, which is an unfair pre-requirement.

Details

I suggest changing to a permissive BSD 3-Clause License for this project. It asserts these requirements for legal reuse and redistribution:

1. Redistributions of source code must retain the above copyright notice, this
   list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
   this list of conditions and the following disclaimer in the documentation
   and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its
   contributors may be used to endorse or promote products derived from
   this software without specific prior written permission.

Outcome

Less reluctance to use this repository for someone else's proprietary project, hopefully more engagement with us as an upstream because it's useful for other people than ourselves too

TeleIRC bridges only deploy from {{ default_version }}

Problem

The Go migration will only install a single Go binary based off the {{ default_version }} listed for TeleIRC, and does not respect individual projects' listed versions to use.

Fix

The compile task needs to be updated to include a dict loop to build a TeleIRC binary based off each project's respective git branch.