Technical debt / possible disaster 

I've been on leave for six months.

Our department knew I would be going on leave years ago. UL never heeded our warnings that having all systems in the hands of one person (me) was a recipe for disaster when that person went on leave / got hit by a bus / etc. They arranged for a part-time IT person w/no lib systems knowledge to get root access to our systems just before I left.

Now it seems that two years of deposits to our repository have disappeared? FUCK.

Technical debt / possible disaster 

The "technical debt" aspect comes into play as I can't help but blame myself for never being able to get certain aspects of system enhancements done before going on leave. Lots of reasons for that, of course, but maybe if I had been able to get that upgrade in place etc this wouldn't have happened.

But then I think about the chronic underresourcing despite all warnings and... maybe this is a (costly) learning opportunity for our admin?

Technical debt / possible disaster 

The question is, though, what lesson are they going to take away: that they need to invest in one more person so there's an actual library & archives systems team?

Or that they don't trust me and they should outsource everything possible so that they can avoid any local dependencies?

Technical debt / possible disaster 

Logged onto the server this morning and found the disks were 100% full. My hypothesis was that the underlying files were still there, but the Solr index ran out of space when it was reindexing in chronological order... thus the missing two years.

So I cleared some space, reindexed, and voila, access to the documents was restored.

Told admin a "disaster was averted, again, but a one-person systems team is a recipe for eventual disaster". We'll see.

Follow

Technical debt / possible disaster 

So much for being on leave...

· · Web · 1 · 0 · 1

Technical debt / possible disaster 

Aside: it looks like the servers haven't had OS-level updates for three months--the last time I dealt with an emergency issue while "on leave". So much for even the most basic level of support

Mood: head in hands

Technical debt / possible disaster 

@platypus Thanks. As much as this particular situation sucks, I have health, family, and a reasonably secure job. All things considered I'm very lucky.

But still... argh.

Technical debt / possible disaster 

@dbs
"Hey boss, it looks like the substitute admin is on actual leave!"

Technical debt / possible disaster 

@dbs
I hope this isn't something we're supposed to be handling (and obvs hit me up asap if it is!)

Technical debt / possible disaster 

@sleslie Nope, running "apt update && apt dist-upgrade" on a regular basis falls under level 1 support in my books. You're all good!

Sign in to participate in the conversation
Mastodon

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!