Learning Linux Towards Security Apprenticeship, Part 2

What’s the next set of concepts I need to learn about in order to advance in my journey towards a security career? Glad you asked! They are the Boot Process, Process Initialization, Process Management & Process Debugging!

In my previous writeup, we covered very fundamental concepts and expertise that you need to develop if you would like pursue a career in Security. We’re going to start doing more practical work in this post.

Reverse engineers need to figure out what the heck is going on in a program or a system. How do they do this? With basic linux troubleshooting skills. In this writeup, we’ll learn about how to learn what is going on in an operating linux system.

These topics can be a little tedious. I urge you to resist the urge to learn “enough to get by” and master these topics.  A large volume of your practical security work will build upon this knowledge.

Learn about the Boot Process

The boot process is the set of activities that initiated when we apply power to a linux system.   An attacker with physical access to a device will succeed if your boot process fails to check for authentic & authorized software.  A boot process that authenticates software reduces the risk physical access attackers.  
The boot process is critical to the security of voting systems, commercial kiosks, mobile devices and IoT devices.  Knowledge of the boot process will prepare you for future discussions involving “secure boot.”

This boot process overview will help establish the foundation you’ll need when you’re ready to learn how Secure Boot works: https://linuxjourney.com/lesson/boot-process-overview

Many IoT devices use uboot. This video does a nice job bringing you up to speed on the basics of uboot: https://youtu.be/INWghYZH3hI

Learn about the “Root of Trust” in IoT. This video is exceptionally accessible and delivers education on a large body of relevant material: https://www.youtube.com/watch?v=h2sdaGsAs7o

Learn about Process Initialization

An operating system launches a a large quantity of processes after initialization. Incident response engineers need to identify how processes launched. You may be responsible for delivering solutions that work after a system reboot. Process Initialization is the set of activities you have to master to perform those tasks. Process initialization in linux has become somewhat controversial in the last few years. Sysv Init is the legacy approach to process initialization in linux. In the last 6 years, systemd has become a dominant solution for initialization lof linux distributions.

You can learn about sysv init & systemd here:  https://linuxjourney.com/lesson/sysv-overview

For an opinionated review of systemd, watch this video: https://youtu.be/o_AIw9bGogo

Learn about Process Management

You will need to diagnose how an application is performing.  You will need to test how a misbehaving application is launched, how much memory, cpu & cache the application is consuming, etc.  https://linuxjourney.com/lesson/monitor-processes-ps-command

Learn about Process Debugging

You have to be able to inspect how a process is performing to identify the effectiveness of it’s security. To do that you need to learn process debugging. You’re ready to start doing real security work after you develop some basic debugging skills.  Debugging is a very large topic.  I’d go through material like this to get a lay of the land for all the tools you could use for debugging.  This is not a comprehensive list:

https://www.thegeekstuff.com/2011/12/linux-performance-monitoring-tools/

Here’s a nice Linux Foundation talk on Debugging Methodologies: https://www.youtube.com/watch?v=1u3pkSULOeM

Julia Evans writes great intro zines for various linux concepts. Julia gave an accessible presentation to strace basics here: https://www.youtube.com/watch?v=4pEHfGKB-OE

You have to learn how to use GDB on applications you didn’t write: https://www.techbeamers.com/how-to-use-gdb-top-debugging-tips/

Learn how to debug an application’s System Calls using strace. https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-work/

Learn how to use strace with GDB: http://hondu.co/blog/write-a-strace-and-gdb

Analyzing tcpdump to wireshark with strace:
https://ops.tips/blog/tcpdump-to-wireshark-in-realtime/

Using strace to figure out how git push over ssh works:
http://kamalmarhubi.com/blog/2015/11/21/using-strace-to-figure-out-how-git-push-over-ssh-works/

Debugging Embedded Devices using GDB: https://www.youtube.com/watch?v=FnfuxDVFcWE

You’ll also need to learn about interprocess communication within Linux done through Dbus. Dbus is a daemon that provides easy, simple IPC communications.
Learn about dbus basics: https://www.freedesktop.org/wiki/Software/dbus/
Systemd and dbus https://youtu.be/-bEzHG2u8XA


In my next post, we’ll start talking about networking. Did you find this nourishing? Something missing? Send me a nudge on twitter or linkedin and we’ll make it even better.

Security Apprenticeship

“What do I need to know how to do in order to pursue a career in security?” Good news! I just happen to have drafted this roadmap of reading material just for you!

In my previous writeup, I provided a roadmap of reading material for anyone who is interested in developing a career in security. The material covered concepts that should be understood by anyone working in the field- but the material didn’t cover how to “do” security. This article summarizes topics & activities that are well understood by anyone doing security work professionally. If you develop mastery of these ideas, you’ll be approaching a point where you can start doing meaningful security work. Unfortunately you won’t be done reading after you’ve finished this writeup. I have at least two other guides in draft states that will help guide you on your path towards pursuing a career in Security. This guide helps you establish your foundational knowledge of techniques for restricting access to a system.

Walking The Security Practitioners Path

If we distill security to its most fundamental concept, security is about the controls that ensure that activities are authorized on a system.  

We can talk about these ideas abstractly, but unless you actually implement these ideas on a working system, your knowledge won’t have much application.  The best way to learn these ideas is by using Linux.  

Linux gives you plenty of experience with programs that don’t work because of security controls.  You’ll have to learn how to debug those problems in order to get the processes working and eventually you’ll develop taste for the correct way to implement those controls.

This next phase of your education covers how to activate a linux system, implement interesting programs like web servers and some basic capabilities for implementing access controls.  My goal in writing this is to provide pointers to well written guides that will help you learn to start “doing” security- which is to say, how to activate controls that protect a system.  In my opinion, mastering these topics should be manadatory- however I should highlight that I know many people in the security industry today who don’t have mastery i all of these topics.  There are areas in here that I’m not as strong as I’d like to be, as well. Everyone who is good at security struggles with the idea that they need to do more reading when time permits.

Learning Linux

Part 1- Well Begun is Half Done– Dip Your Toes into Linux.

Linux: TL;DR:  https://linuxjourney.com/

Learn what Linux is and it’s history.  You’ll want to learn about the basic purpose of linux if you have no experience with the Operating system.  Some day, you’ll need to learn about important related concepts like Posix (https://en.wikipedia.org/wiki/POSIX) and Ansi C (https://en.wikipedia.org/wiki/ANSI_C).  For now, let’s just learn some history: https://linuxjourney.com/lesson/linux-history

Learn about the Shell (aka the command line) on Linux.   Learning the shell is the Computer Science equivalent of learning to walk or bike.  You have an operating system or applications you want to interact with?  Generally you will be working within a shell.  A shell is ‘just’ a program that gives you the ability to use a keyboard to interact with the Operating System.  Some day you will need to write shell scripts to automate tedious tasks or filter large amounts of data through scripting, which is a way of executing a series of commands that will execute in some order based on whatever conditions you chose to define.  You’ll learn about “environment variables” like PATH and PYTHONPATH, which are variable names with values you’ll modify in order to make new programs or libraries accessible from any filesystem location while you’re in the shell.  The shell is your foundation for working with a linux machine.  https://linuxjourney.com/lesson/the-shell. Alternatively, this tutorial teaches you how to get around the shell.

Learn about User Accounts & Groups.  This helps you understand the foundational framework of access controls on computers.  When you need to protect data from access by an unauthorized party, you’ll need to use account & group management concepts.  Learning about accounts & groups will help you understand important concepts like what the “root” account’s purpose is, what the “wheel,” “www-data,” and “nobody” groups purpose are and more.  As you develop your security skills you’ll eventually need to learn about how hackers elevate privilege.  Learning accounts and groups his will give you a foundation when we are ready to learn about “privilege escalation” https://linuxjourney.com/lesson/users-and-groups

Learn about File Permissions.  When you are ready to actually protect that data from unauthorized access, you’ll have to read & evaluate the correctness of filesystem permissions.  If you have a file on a system with multiple users, you need to learn how to control who can access it and define the level of access (read, write, executable, SUID, etc).  Again- some day you will be responsible for helping make sure sensitive files are only accessible by the correct people- and that those people have the right level of access.  You need to learn file permissions to perform this task.  One way to know if you’ve mastered this topic is to evaluate if you can propose accurate Access Control Permissions on a web directory.  Make one version that’s dangerous, and one that’s safe.  Be able to explain why they’re dangerous, and how they could be exploited.  Start your journey here:  https://linuxjourney.com/lesson/file-permissions

Learn about the “Filesystem.”  A file system is the logical structure where you store “files.”  An operating system without the ability to process files is of limited utility.  In the windows world, we have drives with names and folders.  In the MacOS world, we have drives with names & folders.  in iOS & Android, there is a file system, but it’s implementation is not as central to the user experience.  In linux, deeply understanding what the directories are under”/” is critically important.  There are “files” in the /proc directory that can tell you important statistics about system performance.  Configuring servers that you’ll install will require an understanding of the /etc, /var and other directories.  If a server is under DDOS attack, you’ll need to understand information about the number of network connections that the system is currently supporting.  You can indirectly use tooling like ifconfig to gather system performance information, or you could just do an ls against /proc/net/dev.  You’ll also need to learn about read only filesystems like squashFS (https://tldp.org/HOWTO/SquashFS-HOWTO/whatis.html). Someday, you’ll need to figure out what is actually happening when you type “ls” into the shell and it somehow enumerates the files that were in your current directory. If you type another random command in, it doesn’t work. Why? How did the OS know which version of “ls” to run? Where does this “ls” binary live anyway? Learning file systems is mandatory.  https://linuxjourney.com/lesson/filesystem-hierarchy

Learn about /dev.  An extremely important design philosophy in Unix is that “everything is a file.”  Hardware is directly accessible through the file system.  Some hardware is represented as file system objects.   Learning about the dev directory will give you important insight into how devices work on the system, which at some point, you may want to tamper with if you aspire to be a hardware hacker.  https://linuxjourney.com/lesson/dev-directory

Learn about the Kernel.  The kernel is the mechanism for controlling hardware.  It is where security policies get enforced.  If the kernel is compromised, all security assumptions break.  You need to understand this important resource conceptually to work in this industry- both for defense & offense.  https://linuxjourney.com/lesson/kernel-overview

If you aspire to penetration testing or red teaming, you need to go further and learn about interacting with the kernel.  

Linux programming interface: https://www.amazon.com/Linux-Programming-Interface-System-Handbook/dp/1593272200

Kernel Hacking (as in MIT definition): https://www.kernel.org/doc/html/latest/kernel-hacking/index.html

Kernel Hacking (as in exploitation): https://github.com/xairy/linux-kernel-exploitation

The Snowball Effect

If you’ve gotten this far, we learned about the Kernel, the Filesystem, Permissions, User accounts and a little hardware. This is a good stopping point- we have a golfball-sized snowball- you basically know the perimeter of the pitch. In my next sections, we’ll take that snowball up to the top of a mountain and give it a nudge: We’ll cover how to make things happen on a Linux system through process management and we’ll learn how to make our linux system talk through the power of IP networking. After that, we’ll start to cover the topics of a security practitioner: reverse engineering, vulnerability discovery, exploitation & remediation and eventually how to protect processes, systems & keys.