Upgrade job definition already exists

This blog post is a little late but I wanted to get it out of my mental queue…

What

A customer had a SharePoint 2010 Patching/Upgrade go horribly wrong.  It was stuck. Couldn’t complete the upgrade, PSConfig failed, couldn’t go forward, couldn’t go back. In fact, it was stuck in that state for weeks (possibly months).  The errors/issues exhibited were:

• In Central Admin, servers identified “upgrade available” but no amount of attempting to upgrade, to whatever CU, would succeed
• In ULS, there was lots of:
  • NeedsUpgrade entries for several service application databases
  • Unexpected events of “Exiting the process because the timer could not be initialized after multiple attempts” and “Upgrade job definition already exists, waiting for the existing upgrade to complete”
  • In short… buggered….

 

There were also countless related Timer Service failures with the following… (your mileage on this one could vary…)

Event ID 7031 occurs 16000+ times (every 3 minutes). %%-2147467259

So What

Well, after quite a bit of Googling, er Bing’ing, I wasn’t getting very far indeed (in fact, if you are reading this post, you are welcome, now buy me a beer…  https://itgroove42.wufoo.com/forms/beer-me/)

But lo and behold, it was telling me exactly what I needed to find/see in the ULS.  It was this bad boy in the Timer Definitions…

 

 

A rogue “Upgrade Job” Timer job that had no schedule or history. Just stuck there, like a father chaperoning (alone and confused) at a Justin Bieber concert. Sucks.

Now What

Well, being the good consultant that I am. Before thwacking away at things, I verified against two other SharePoint environments whether such a timer job is normally there and I had somehow not noticed it before. Nope, this bad boy gets created during upgrades and I gather gets nuked at the end before anyone is the wiser. I guess in this instance, it was lost in spaaaaaaaccceeeee…

Anyways, I nuked that bad boy (clicked delete) and restarted the Timer Service.  I had the ULS View open at the time and it lit up like a Christmas tree trying to make good on other things it was jammed up on. In short order, I was back in business and able to go back to patching/PSConfig’ing the environment

Whew. Hope it helps.  Make this post worthwhile for me to share again next time too.. https://itgroove42.wufoo.com/forms/beer-me