Next Door to Derpton – When Your Fellow DBA is a Danger to Databases (Dear SQL DBA)

What do you do when your fellow DBA is a ticking time-bomb of bad decisions, waiting to explode your production environment?

Note: This is a “listen-able” video. You can also listen to this as a podcast – learn how at littlekendra.com/dearsqldba.

Here’s Today’s Question

What do I do with a co-worker (who claims to have 20 years being a DBA) who puts all the production databases into Simple recovery mode?

Sincerely,

Next Door to Derpton

This is a Tough One…

In this case, we’re assuming that SIMPLE recovery model isn’t appropriate for all those databases— and that losing all the data since the last full (or full + diff) backup might be big trouble for the business.

It’s a difficult situation when one of your peers makes decisions that you feel risk the availability and safety of the data. Going deeper, it’s tough being on a team with someone who you feel doesn’t have the knowledge and skills to do their job.

Especially if they might make more money than you do.

You Need Protection, and Change Management is that Protection

The biggest problem is that your data may be at risk. You need to stabilize the configuration of your environment, and make sure that changes to the configuration go through a good review and approval process.

This may sound like a drag, but it protects you as well. We all have those times where something that seems like a good idea backfires on us.

If you don’t have Change Management, you need to become its champion. There are a lot of ways you can champion this for the sake of good process, and management typically loves it.

If you do have Change Management, your mission is to make sure it’s being used well, and that when changes go wrong, you’re finding root cause.

Be Careful Spinning the Wheel of Blame

Should you tell your boss that your coworker doesn’t know their transaction log from their tempdb

Usually, no.

If peer review is a part of your work system, it’s OK to be honest during that peer review framework. Make sure you’re being constructive.

In that case, pretend it was you: you’d want to know the extent of where you needed to improve, but you wouldn’t want your nose rubbed in it.

If your boss asks you what your impression is of your coworker’s skills in a private conversation, think through specific changes that have gone wrong and mention those incidents. Request that your boss keep your comments confidential.

Outside of private conversations with the team manager, change the subject. You’re a team. Team dynamics that turn against one team member are bad for the whole team.

If your team is having problems because of misconfigurations and changes that have gone wrong, look through those changes and make recommendations to processes to fix those.

  • Better change review
  • Better adherence to using change control
  • Improving documentation on how to do things  / breaking down “knowledge silos”

It’s also OK to be honest about areas where you believe your team needs more training, but talk generally about the team.

It takes really hard work to stay positive and keep it from getting personal in this situation, but it’s absolutely the best thing you can do.

It’s bad to have a coworker who lacks skills and may put your environment at risk. It’s even worse to have them believe you’re out to get them!

What if Your Coworker Regularly Goes Off the Ranch and Doesn’t Use Change Control?

Don’t cover for them.

Ask them about it first to make sure there wasn’t an emergency change request you’re unaware of for the change, but be honest about what happened when you’re asked.

In other words, treat them as an equal and a grown-up.

Sometimes in this situation, people sugar coat things or cover for the person who makes mistakes. You need to treat them as an adult though.

If you made mistakes, you would own up to what happened and work to not do it again, right? It’s just about respectfully allowing others to own their actions.

Mindset: Focus on Building Your Own Skills

It’s hard to stay positive in this situation. Your mindset is critical to navigating this successfully without having it drag you down.

As you grow your own skills, you’re likely to work with Junior DBAs more and more.

You’ll need to build strong processes, documentation, and change control to help them succeed.

After working with a peer with those issues, leading Junior DBAs will seem easy, so this is awesome training for a senior level position!

As often as you can, focus on your own learning and your ability to build resilient processes that help people make the right choices (and allow every change to get review and input).
Because after all, that’s good for you at 3 am when the pager goes off, too.

Previous Post
Merging Boundary Points: Does a Changing Partition_Number Indicate Data Movement?
Next Post
Target Recovery Interval and Indirect Checkpoint – New Default of 60 Seconds in SQL Server 2016

Related Posts

11 Comments. Leave new

  • Anon for obvious reason
    June 9, 2016 8:39 am

    What if your boss is the derp?

    Reply
    • #AskingForAFriend

      The answer to this one varies. I’ve seen some situations where the boss was great at being a boss, but just couldn’t let go of the technical stuff. And being a boss takes up most of their week, so as time passes they get rustier and rustier and eventually start messing stuff up by accident. With those folks, having a private, “you’re a great boss and your time is valuable, maybe we should get you out of incident work and leverage you for escalations” conversation frequently works. Because it really doesn’t make a lot of sense for the boss to be managing incidents or deploying changes.

      If your boss isn’t a great boss for you AND is causing technical problems, then you can tighten up change management as best as you can, but there’s not a lot of great options for “managing up” there. I’d keep calm, but stay aware of options to exit that team should conditions worsen.

      Reply
  • Hello, Mister!

    Who’s a good boy?!

    Who’s a good boy?!

    Reply
    • He’s in the same spot today. I just turned and looked at him, and he raised a single eyebrow, just enough for him to tell if it was time to go for a walk or not. 🙂

      Reply
  • Hey wait a second, most of my databases are using the SIMPLE recovery model. Our load is heavily BI oriented, with nightly ETL then back ups. These databases only get read during the day and SIMPLE recovery model seemed the most appropriate. That being said, our few transactional and CRM database are most definitely using the FULL recovery model and have log backups throughout the day per the business’ RPO/RTO guidelines. Does that make me a derp or not a derp?

    Reply
    • I *love* the simple recovery model when it’s appropriate for availability and data loss!

      I made a comment in the recording that we’re assuming some of the databases in question can’t lose data since the last full backup (I should have said full or full + diff), but I think I didn’t make that clear in the show notes. Updating now, thanks!

      Reply
  • “It’s bad to have a coworker who lacks skills and may put your environment at risk. It’s even worse to have them believe you’re out to get them!”

    Disagree.

    As a DBA, my primary duty is to protect data. If a colleague had repeatedly endangered data, or risked losing access to the data, then I would, if a quiet word failed to change things, speak to my manager about it. It’s not school and a workplace team is not playmates in the classroom where telling tales is, traditionally, culturally wrong; it’s business and that means making adult decisions. Hushing-up such behaviour is also, assuming one works for a company that is governed by data protection legislation, derelict, and could lead to uncomfortable conversations later on, if things were to get legal.

    Reply
    • Hey Richard!

      So you say, “If a colleague had repeatedly endangered data, or risked losing access to the data, then I would, if a quiet word failed to change things, speak to my manager about it.”

      I absolutely agree that if things go wrong in the production environment, the manager should know about it. But I believe that should be happening regularly and in a way that doesn’t require individual teammates to make private reports to the manager.

      My suggestion is essentially that championing a good change process provides a structure for everyone to be accountable. Managers usually attend weekly change meetings to approve changes and hear how past changes went. This is actually great for managers, because they need to be aware of what went wrong, as well as what has been successful in their teams.

      I’ve seen several team situations as a consultant where one or more DBAs were very judgmental of the skills of other DBAs on their team, and expressed that to their manager. In these cases, the manager brought me in as a technical expert, and explained the situation ahead of time. What I found in each case was no real change process, and a team who could hardly talk to one another. The team member who complained to the manager hadn’t made the situation better at all, it had stirred up a lot of drama which distracted everyone and broke down their ability to get things done. My main concern is helping people avoid that.

      Reply
  • […] Kendra Little answers a user question about a co-worker who puts all prod databases into Simple reco…: […]

    Reply
  • The maximal amount of data that may be lost during a disaster is a business decision arrived at the highest level. It is one of the first things that I ask when a new DB goes into production. The more frequent the log backups, the heavier the load on the system, which usually translates into cost. High Availability has its costs as well. If it is, say, an hour, then that becomes our log-backup frequency and the DB goes into Full recovery mode.

    All of our DBs receive a full backup nightly. Should the answer come back from upper management that they are happy with more than a days’ data loss in the event of a disaster, then we’ll set the DB to Simple and be happy with the nightly backups. As a consequence, all of our DBs have are in full recovery mode and have regular log backups.

    We have sandbox DBs that are for us DBAs. They are backed up nightly. We are happy to set them in Simple recovery mode. Everything I need is either in a script or in a cloned restore of the relevant prod-DB.

    Reply
    • “The more frequent the log backups, the heavier the load on the system, which usually translates into cost.”

      Nope! Log backups only back up what’s happened since the previous log backup. Doing more frequent log backups typically doesn’t increase load on a system, and frequent log backups generally aren’t a performance concern.

      There’s always an exception – people who load thousands of tiny databases on a single instance of SQL Server can have challenges running any operation against each database, which is an inherent problem with that architecture.

      For many environments, running transaction log backups every 1 minute isn’t a problem if that’s required. The backups are lightning fast.

      I will say that your example of log backups every one hour is a red flag to me. I can only think of two times when I’ve talked to business owners and asked, “is it OK to lose more than an hour’s worth of data?” and they said yes. Typically people either want to lose almost none or are comfortable with a day’s loss/reloading, it’s pretty rare to find things in between. Possibly you’re in that small sample of people, but it doesn’t hurt to make sure.

      Reply

Leave a Reply to Tammy! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Menu