Sysadmin (or devops), complexity and impostor sindrome

I read on and off reddit.com/r/sysadmin since years.

There a lot of people mention the impostor syndrome. Another problem is the fear that one knows very a little and there is still a lot to learn, as what is needed to know seems manageable. One may think "hey cannot be that hard, those are a limited amount of technologies that we use, I will master them soon, but it seems to take longer than expected".

Well I had some of those thoughts too. Side note for me, I should write about being perfectionist too, as the wikipedia article mentions it. A very detrimental - and at the same time arrogant - attitude. This because one doesn't recognize that wanting to be perfectionist means the wish to make a error free solution and at the same time wishing to reach that solution in a practical time. First history shows that everything non trivial made by men can be somewhen improved and, second, that non trivial solutions with a lot of iterations that took away the majority of problems took decades (if not more) to be designed. Or in other words, if we would apply perfectionism to the water we want to drink, taking it from a pond (that is, taking a very imperfect initial situation) and wishing to make it perfect for our taste before to drink it, we will be long dead before hoping to drink even a drop of it. This is an example to say, perfectionism is detrimental. Enough of a digression.

So what does a sysadmin (in IT) and in part their variations (system engineer, system reliability engineer, devops, etc..) do? They take solutions mostly developed by others, they combine them to offert to end user services to be used. With a rough analogy, they combine construction elements to make roads and offert to end user use roads and vehicles (say like public transport).

Now at first it may seem that the complexity is mostly related to building those construction parts, or the solution to be combined to offer the services. Sometimes it is so. Surely developing a webserver or an operating system or a continous integration tool or a configuration management tool is no easy feat. The amount of configuration management tools out there is much smaller than the amount of teams that use them. Thus one may think that the field of sysadmin can be somehow quickly mastered.

Maybe. The problem is that even narrowing the domain to <Ok you have this list of limited technologies, you "just" need to configure them and let them interconnect and you are good to go> , it may make the domain easier but by no means easy to master.

To explain what I mean I will make an analogy with Chess. Yes I know you may say "you are really fixed with tha game, it is just a game!".
I don't agree. It is a very narrow domain of activity, compared to life, that is still impossible to master for humans. Even just considering the amount of not low effort games, one still has a number that is practically infinite for a human to digest. More that that is that chess is played in a competitive way, studied and recorded since centuries (games are recorded but also the life of the best players). This allows us to appreciate how difficult it is and the evolution of performance of some players. In short, thanks to the growing amount of data about chess and chess competitions, it is a wonderful laboratory to get some insights about human intellectual performances. Not in terms of IQ (that is another topic), rather in terms os a certain type of problem solving, how long it takes to master something far from trivial and so on. There are other domains like chess, but chess is good enough for a comparison.

Said that, how sysadmin/devops and the business of "combining elements to provide services" compares to chess?
Well in chess there are few pieces, a limited board, and still one has a vast amount of sensible combinations of them. In a sysadmin environment one has a limited number of technologies, say a webserver, an operating system, the applications on the operating system, the backend service, the database, the networks and so on; a limited number of properties to set for each technology, for example the virtual host in the webserver, the IPs, routers and firewall for the network, the backend port for the backend service, the location on the filesystem (or the object storage, and so on); and has to combine them to make them work in a satisfactory way if not - when possible - in a way that is among the best one. The latter is particularly important when the resources are limited compared to the demand of the service, thus when wasting resources is very critical.

This is the case in which, in the sysadmin field, as well as in chess, combinatorics comes to play. Even considering a fixed number of configuration properties of the components of a system managed by the sysadmin, say for example: all the possible directives that can be configured in a webserver, and if those can be repeated, like virtual hosts or rewrite rules, let's assume to limit them to a dozen at best. Even considering "only" those, one has hundreds, if not more, attributes to configure, each with possibly a handful of values (if not more). That is, together the possible coniguration for the webserver quickly become vast and the sysadmin should not only avoid those that let the webserver not work, should also identify those that bring a good performance.

And this was only one webserver, with restrictive assumpions. Then there is the backend service, the database, the network and so on and so forth. All those elements have tens or hundreds attributes that can be configured, and each could accept several, if not more, values. All combined together - as the system as to work as a whole, and thus things have to be combined, the possible confgurations that a sysadmin can input explode quickly to a number that easily eclipses the chess combinations.

Now, chess is impossible to master, but even being among the best players in the world is a feat only a handful can do after years of continous effort. There are hundreds of million of people that know the rules of chess, yet only about 2000 people could be considered among the top players (grandmasters), and only about 50 are among the elite players (super grandmasters).

Imagine what it takes to master the combination of technologies in the field of a sysadmin! For sure a sane person should know that he knows a litle. There is a vastness to explore and master, there is always something new to notice. Or a configuration may fit a case, but not another. (Like in chess a move, that is, a configuration on the board, may be strong in a certain situation but weak in another)

Let's make a very small example, to see how quickly things can explode in complexity.

Let's take a system with a webserver, the firewall and network, containers in the backend, 2 types of backend services (as containers) that can be configured as well, one database (that can be configured), 4 system services to configure.

  • Let's say that the webserver has some 30 attributes to configure (actually they can be much more, like rewrite rules, log, etc..) each with 5 possible good values.
  • The firewall and the network have 5 attributes to configure, with 5 possible good values.
  • The two services in containers again 5 attributes and 5 possible good values each.
  • The database has 10 attributes, each with 5 possible good values.
  • The 4 system services have each 3 attributes, with 5 possible good values.

In total there are 67 attributes to configure, each with 5 possible values. The possible combinations are 5^67 or 67'762'635'780'344'027'125'465'800'054'371'356'964'111'328'125 combinations. Even trying a combination each second, one would need: 2'148'739'084'866'312'377'139'326'485'742'369'259'389 years to try them all.

A huge search space. Indeed many sysadmin are happy with "good enough" solutions, not with the best ones. Getting the best ones is no easy feat and requires a lot of experience, understanding and skill. And this was just a mini setup. In real life the situation could be much richer in attributes to configure, and I ignored completely scripts or attributes with potentially extremely large possibilities, like rewrite rules or attributes that accepts thousand of possible numerical values.

For this one has to accept to know to know a little. The field, although it seems trivial, offers a vast challenge if one wants to understand or predict how the various configuration works together. It is like a chess player trying to find a good move in always new situations.

It is challenging, it is interesting, although at first it seems a very limited problem. As in chess, there are only 32 pieces and 64 squares, how hard can it be, right? In the sysadmin field, if the attributes are the pieces and the possible values the squares on the board, then one playes a extremely large variant of chess. That is very interesting, so let's not be discouraged! Let's try our best and document for those that will follow!

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License