STUCK in a an Authentication Loop being Pwned
By Tony DePrato | Follow me on Twitter @tdeprato
There are two things you never want to happen. #1 – You never want anyone to gain access to the root user or super user account of any system or piece of hardware. Even an air purifier or printer can be turned against your network if this happens. #2 – You never want an application to be stuck in an Authentication Loop where the password it is trying to use is no longer valid. On Friday the 13th, 2016, this is what happened to me and my team.
Authentication Loop
Imagine a two year old that has decided to say the same word over and over all day. Maybe something like, “No!” or “Nope”, or even “DogDogDog”. After about four or five hours, most people would be near the edge of insanity. This is the same thing that can happen to a database when a program sends the wrong password to the database a hundred times in a minute.
The database is like, “Hey! This is wrong. Stop sending me this password!” Then the database decides to make a note every time it happens. In about 60 seconds (in my case) that equates, or equated, to about twenty notes. This is roughly 1200 notes in a minute.
The database is not happy about working so hard, so it locks the application out, and no longer allows the application, or the IP address from where that application lives, to try to connect anymore. It removes all VIP privileges, and sends that application back outside to wait in line with the other services who may or may not be allowed to join the club later.
Being PWNED as a Team
Skipping all the technical details, my team and I knew what was causing the problem. We followed the DIY steps on the support forum only to find that none of them worked. We also found via forum comments some people never gained control of the problem and waited days for a third party to rescue them. It was pretty hopeless. Hopeless is a term used in education when a program or service is about to be the focus of 1000 parents doing online registration, and that program or service is offline. It was hopeless.
Anyway, we were PWNED or Owned or being controlled by someone’s badly written software with a broken process for updating it’s login credentials. We were under its control, even though it only really had the same control my cat does over her need to constantly go outside and back inside, and then back outside.
I can say though, it is much better to have a team approach to solving problems, even if some people on the team are asking seemingly unrelated questions. I noticed on the discussion forums that people who seemed to never solve the issue, were lone tech people in a basement somewhere trying to speak the illogical cat language of the faulty program. I was lucky to have a person next to me, and a person on email, standing by for support. I also had two people to manage the complaints that were streaming in. I cannot imagine working on the technology and emailing the frustrated families at the same time. All in, 5 people.
Why Those in Education Need to Care about this Story
This story is not about technology, or a cat. This is about a department that had a plan for disasters. A department that had a team which had practice responding to problems, and containing the damage. Forget prevention. When the unknown happens, life is about containment. Prevention is too late when the problem occurred in the past. Focusing on prevention in the middle of a crisis is a fools errand.
Anyone who is in a leadership role needs to sit down with their teams a few times a semester, and instead of reviewing jobs and complaining about people, the team needs to imagine disaster. The team needs to imagine multiple lose-lose scenarios where there may not be a positive outcome.
Working through this thought exercise is the best way to prepare for the unknown. When things do happen, a team who has practiced is going to be in the correct mindset. Eventually they will calm down and the panic will subside. Once each member knows they are supported and have a job to do (and not all jobs to do) then, and only then, is a possible solution going to appear.
In education there is too much time spent focusing on the positives, not being critical, and not actually discussing the worst case scenarios. Scenarios are often specific to each operational department. Although a school might prepare for a fire or earthquake, they probably are not preparing for an angry parent attacking the institution with a onslaught of paid social media. That can be just as damaging as a fire, and actually worse, since there is no such thing as social media smackdown insurance.
Departments of ONE are a Bad Idea
Never have a department or team of ONE. If there is a department of ONE add someone to that department as a liaison or assistant. Any administrator can be an assistant to a department. Administrators often can help improve communication in an emergency and bring other resources to bear. Leading in one area, should not exclude people from being a participant in another area.
A department of ONE, regardless of their skill level, cannot work and communicate at the same time. Both functions are important. School leadership needs to audit all departments and support functions(groups) to ensure no one is standing alone in an emergency.
Had I been alone on Friday the 13th, my only solution would have been a broadcast email to parents telling them the system was offline until further notice. Imagine reading that message. As a tuition paying member of the community, how would you feel?
It is going to happen. You and your department, or campus, or team will be PWNED. Prepare. Think. Be paranoid. When the initial noise has subsided and the problem is clear attack, and work, and if you fail do it together. If nothing else, it makes for a better story.
Tony DePrato
Source: IT Babble Blog and Podcast
You must be logged in to post a comment.