posted on 2015-08-09 10:08:14
It just so happened that a colleague need to get serious linux skills for a new job... fast. Timeline is like three months.
What to do about that?
I have used linux for years, what have things were the most important during all this time?
This is the problem at hand, and there might be a series of posts as a rant and some kind of exploratory research on what I'd learn in what order with all my knowledge today, if I had to relearn things from scratch.
The underlying theme is to get the basics right and literally all the rest will follow.
Oh, and forget what university told you, this is practice, baby. University is worth jack, you know less than an apprentice at an ISP once you are finished, no matter what your master's average was.
What are essential skills in 2015? How about these as a braindump, importance-wise roughly sorted in that order:
This is what I'd consider essential as of this moment, likely this is not perfect. :)
To learn things, knowing how to use a virtualization technology like virtual box is key, but from the start you will not know what you are doing anyway. Get a grip on the other things first, you will see it is way easier after the basics.
All in all this will give a solid base to start from, and give a proper perspective for working with 'this so-called internet'.
For real learning purposes and the best bang-for-buck, don't bother with windows.
Windows may be a neccessity as you will have to have to work with it, but in the long run you will get much more out of unix-based knowledge. It is used in appliances, switches, stable servers, 'real' clusters. There are things you just cannot build with microsoft stuff directly, from what my windows colleagues explained to me.
Read that? You. Simply. Cannot.
If you doubt that, build a real multinode cluster without paying your ass off for some crappy VMWare stuff where the licensing fees will eat your lunch, revenue-wise.
Using linux as a base to get this knowledge is the easiest way, just know there are the BSD's and other unices out there, too.
Even if you have the money for VMware licenses, it's 'enterprise'.
"ENTERPRISE, FUCK YEAH."
This translates to 'build a product, get market share, earn $$$ for crappy software and use your vendor-lock-in to quench as much money out of your loyal customers as you can'. All the while using these customers as guinea pigs, as software development is expensive and rigorous testing cannot be covered by your budget.
Using open source software I get the absolute same results without paying shitloads of money. This is why redhat thrives on being successful with just providing support, even when CentOS serves the same functionality as RHEL.
Also having an accessible operating system at hand for your hypervisor or 'appliance' ('appliance' is a swearword for me, TBH) instead of some crappy busybox is immensely useful. Don't let you tell otherwise from the naysayers.
But uptime requirements become more harsh nowadays, and cluster solutions by most vendors are just not there. (I have bled a lot with a SOPHOS cluster in the past. Discussions with their support were absolute crap, too. But really all vendors are the same, some more, some less.)
Once you have an 'appliance' (which is just a firewall+IDS/IPS external and in front of your production systems), you are covered for the 80% of use cases. Which is nice, if you just need a single box for securing a small network.
Not wanting to run a custom-built OS on generic hardware, but buying a dedicated box?
Don't bother, it costs a lot of time, will not work reliably, is just expensive in every way.
I was going to write more here on some real-life examples with examples, but this will be stuff for another posting.
Once having an idea how things work, this knowledge can be adapted easily to windows systems from my experience, whereas windows users often have really hard times with the CLI. But often CLI is all you have, and I dimly remember microsoft's hyper-v has soon (if not yet already) an install mode where you only have a console, and no gui anymore. This in combination of the advent of the windows powershell just screams, 'get comfortable on the commandline'. Powershell is just bloated, unintuitive and complex from my perspective, but at least you can automate things an order of magnitude better with it. Also SSH finds it way into windows.
Man might see a pattern here. :)
Out of neccessity, a word from the wise:
To get some perspective for the linux fanboys out there: Even linux is not the pinnacle of everything. Lots of doubled functionality and applications doing the same things and you have to know like three applications where a good single one would cut it, too. Open-source driven without a real single paying customer behind it, development-wise it is like PHP. There sure are companies investing money in the kernel and userland development, but there are just to many directions at once. It is not like a single concentrated effort behind everything, spreading the development power thin.
Do you think it is just a coincidence Netflix and Whatsapp run on BSD systems?
But for things beside the standard use cases, BSD's are not the wisest choice, too.
If you need esoteric stuff, linux is faster up and running, has the better drivers (or has at least even drivers at all) and thus runs on much more hardware more easily (NetBSD? whatever.).
At the end of the day there exist simply way more userland tools on which more work is done.
It is not just about the core system, you BSD guys.
Just compare linux'
top with bsd'
top, functionality wise.
I know, that this just a question of manpower. But how about shifting the BSD focus from servers to the becoming THE operating system for the internet of things? Gaming consoles are already more likely bsd-based, for example.
There are several layers, in short:
Starting with hardware or software raid setups, giving you redundancy. Over the different discs (HDD vs. SSD and the available types, block size stuff, and interfaces like SATA vs. SAS). To network technologies, but these are not of interest in a 'basic course', just know you can access storage via networks, too.
Software vs. fake raids vs. hardware raids. What you usually use, what types exist with which tradeoffs. How much you can rely upon them really.
What happens when things break? What issues can arise, and how can these show up?
This is basic essential stuff. It's needed for redundancy, and HARDWARE DOES FAIL. Period.
You need it, and you need to know what you are doing.
There are BIOS based systems, but the transition it currently to UEFI. The hardware brain of the computer is only indirectly linked to you partitioning your disks, but there are reasons MBR and GPT's exist. Also the knowledge from how your system boots from the last section comes handy here.
Also you have to know about the almighty LVM, which lets you do things you would not be able to do otherwise.
How the actual files are stored on disk, which is just a stream of magnetic (or other techniques) information, what are the differences. How you can use these, what are advantages of one over the other.
V(X)LANs, LACP, QoS, traffic shaping are rather less important concepts but you should have heard at least what they are.
From the datacenter at work to your router at home and in the 'internets of things' (tm) within the next 5 to 10 years.
Sure, the networks are depleted, but things will keep on for a little longer while, NAT will help until larger adoption. IP v6 will not see huge adoption rates unless the carriers and telcos agree on a switch and consumer hardware sees nationwide rollouts. As long as the old consumer routers do not speak ipv6, there is no point in doing a grand scale switch. Adoption rates in every country of the world were usually way below 10 percent the last time I checked.
It is nice to know that ipv6 exists and getting a dual stack setup up and running is nice, but not something to learn if you have only three months.
Concepts are universal, and you can try them out easily at home. (Exception here are the VLAN's, you need a switch being able to do these as well as working NIC's.)
No need to go complete lowlevel, but you should know what the difference between network devices is and what a how a broadcast domain is different from a collision domain. No need to know what multiplexing really is, just know ethernet exists and this is what switching is about, whereas routing is the "ip stuff".
Linux will help there, as the kernel can do a lot of things so you can play with networks.
In 30% of all cases when things break, it's DNS stuff. (At least that how it feels for me, the guess may be off.)
It's easy, it's simple. And sometimes people running a web agency for over 10 years are too stupid to set up an A and a PTR record properly?
You got to be fucking kidding me, but I am not making this up. It's easy, just noone bothers ever to tell new people how to do things right.
Virtual private networking.
Three words, endless hours of unfruitful troubleshooting and disconnectivity, if you are lacking your network basics. Still essential in everyday business work.
When it is just so simple, if you have roughly an idea what you are doing with networks. For openVPN sprinkle in a 'little' PKI / SSL/TLS certificates knowledge.
But cert knowledge is an absolute MUST in the long run, no matter what you do.
You have to blindly know how to use
openssl, how certificate files can and have to look, how to they are actually created.
There exist several layers of virtualization, and there is no really good differentiation out there between some of them. But they exist and are important as they help you a lot with your work, they let you try things without having to reinstall servers completely, are just faster then playing with regular hardware and thus enable way faster feedback loops.
Using snapshots is just damn easy:
Need to try an update, have no test environment at hand?
If it goes wrong you are in deep shit?
Operating system virtualization has got you covered.
There also exists storage virtualization, like DRBD, which is essential for budget clustering without a dedicated shared storage. It's basically a RAID 1 setup over a network connection.
Of course you can get an EMC or dothill storage or whatever. But that is spending $$$ again, and often you do not need the extra performance through premium hardware (except for virtualization cluster environments) or just cannot afford it.. These SATA 6g platters don't pay themselves, and waiting weeks for a new harddisk due to delivery issues does not help your damaged RAID or your nerves. And when not using original hardware you may void your warranty, and are just as bad off if you built the box yourself in the first place..............
Hand in hand with virtualization go storages and storage technologies.
Be it local or network storage.
And non-local storages need network connections.
And network connections need to be setup on the operating system.
This is the full circle.
Once you need to get an understanding on clusters, simply build on all of the foundations above and be amazed how easy all this falls into place.
And be left wondering why others seem to have such a hard time with it, or cannot seem to know where to start fixing when things break down.
This was written in a single session, lets see how it holds up over the next three months.
View posts from 2017-05, 2017-04, 2017-03, 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10