Dev-Ops Tips & Tracks

发表于 分类于 阅读次数： 本文字数： 33k 阅读时长 ≈ 30 分钟

DevOps BOOTCAMP

DevOps

  • Pipeline
    1
    Collaborate -> Build -> Test -> Deploy -> Run -> Configure -> Monitor
  • Tradition: Development vs Operations
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    # Development
      - programming languages
      - test frameworks
      - databases
      - version control

    # Operations
      - OS, mostly Linux
      - command-line
      - scripting
      - monitoring tools
  • DevOps culture
    1
    2
    1. DevOps was just a way of working between DEV's and OP's
    2. Common language to communicate
  • Waterfall vs. Agile
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    # Waterfall: 瀑布开发
      > ineffective process
      > over time new requirements may arise
      > many places of failure and miscommunication
      - Requirements:
        - plan everything beforehend
      - Development:
        - developers code complete app
      - Testing:
        - testing after everything has developed
      - Operations:
        - huge preparation

    # Agile: 敏捷开发
      > speed of development, testing and development cycles
      > each feature gets tested, deployed
      > immediate feedback
      > fast development and deployment process
      - Continuous Integration:
      - Continuous Delivery:

Operating Systems

  • OS & Linux
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    # Intro Operating Systems
      - Hardware of Computer:
        > CPU + Memory + Storage + I/O devices
      - Operating System:
        - Translator
        - Manages Resources
          - CPU Management:
          - Memory Management (RAM = Rapid Access Memory)
            - Memory Swape (Swap-out/Swap-in)
          - Storage Management:
            - Manage File System:
              - stored in a structured way
              - in Unix systems: tree file system
              - in Windows OS: multiple root folders (Called drives, uses backslashes for paths)
          - Management I/O Devices
          - Security & Networking
            - Security:
              - managing users and permissions
            - Networking:
        - Isolates(隔离) content of application
      - Operating system Components:
        - kernel:
          - kernel is a program
          - Consists of device drivers, dispatcher, scheduler, File System etc.
      - POSIX: Portable Operating System Interface
      - Linux and MacOS both POSIX compliant
  • Windows & Linux CLI
Windows Linux Description
dir ls -l Directory Listing
ren mv rename a files
copy cp Copying a file
move mv Moving a file
cls clear Clear Screen
del rm Delete file
chdir pwd returns your current directory location
time date displays the time
cd cd Change the current directory
md mkdir To create a new directory/folder
echo echo To print something on the screen

  • Virtualization虚拟化

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    # VirtualBox:
      > VirtualBox takes hardware resources from Host OS.
      > Creates virtual CPU, virtual RAM, virtual storage for each virtual machine.
      > Hardware resources are shared

    # Bare Metal(for server): Hardware -> Hypervisor -> Guest OS
      - wmware ESXi
      - Microsoft Hyper-v

      - Efficient usage of hardware resources
        - use all the resources of performant big server
        - Users can choose any resources combinations
      - Abstraction of the Operating System from the hardware
        - Secure very easily
        - Portable
        - Not dependent on physical server
    # Sharing things is not good for isolation

  • Linux File System

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    > Everything in Linux is a File
    # Linux File System
      - hierarchical tree structure
      - 1 root folder
      - multiple users on computer
      - each user has its own space
      - each user can have own configurations
      - programs installed system wide, are avaiable for all users

      ➜ tree / -L 1
      /
      ├── bin -> usr/bin      -- executables for most essential user commands
      ├── boot                -- contains files required for booting (Read Only Folders)
      ├── core
      ├── dev                 -- location of device files, like webcam, keyboard, hard drive etc.
      ├── etc                 -- place where configuration for system-wide applications is stored
      ├── home
      ├── k8s_data
      ├── lib -> usr/lib      -- Essential shared libraries that executables from /bin or /sbin use
      ├── lost+found
      ├── media               -- contains subdirectories, where removable media devices inserted into the computer are mounted.
      ├── mnt                 -- historically, sys admins mounted temporary file systems there
      ├── opt                 -- third-party program you install
      ├── proc
      ├── root
      ├── run
      ├── sbin -> usr/sbin    -- system binaries: essential system binaries programs that admin would use (need superuser privilige)
      ├── snap
      ├── srv
      ├── sys
      ├── tmp                 -- temporary resources required for some process, kept here temporarily
      ├── usr                 -- this was used for user home directories
        ├── bin
        ├── config
        ├── games
        ├── include
        ├── lib
        ├── libexec
        ├── local             -- Programs that you install on the computer; will be avaiable for all users on the computer
        ├── sbin
        ├── share
        └── src
      └── var                 -- contains files to which the system writes data
        ├── backups
        ├── cache             -- contains cached data from application programs
        ├── crash
        ├── lib
        ├── local
        ├── lock -> /run/lock
        ├── log               -- contains log files
        ├── mail
        ├── opt
        ├── run -> /run
        ├── snap
        ├── spool
        ├── tmp
        └── www
      21 directories, 1 file

      - /usr/local vs. /opt
        - /usr/local:
          - programs, which split its components
        - /opt:
          - programs, which NOT split its components

    # Hidden Files
      - ls primarily used to help prevent important data from being accientally deleted
      - Automatically generated by programs or operating system
      - File name starts with a dot
      - In UNIX also called "dotfiles"

    # Windows File System
      - multiple root folders
      - A,B: Removable Disks
      - C: Local Disk - Internal hard drive

  • Basic Linux Command

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    # GUI vs. CLI
      - GUI = A graphical user interface
      - CLI = Command Line Interface

    ubuntu in ~ at k8s-node1 via 🐹 v1.16.5 via 🐍 3.8.6
    ➜
      - ubuntu: user name
      - k8s-node1: computer name
      - ~: home directory
      - $: represents regular user
      - #: sign for root user

    # Folder Operations
      - pwd: Show current directory
      - ls: List folders and files
        - ls -a = Show all (including hidden)
      - cd: Change directory to [dir]
      - mkdir: Make directory [dirname]

    # File Operations
      - touch [filename]: Create [filename]
      - rm [filename]: Delete [filename]
        - rm -r [dirname]: Delete a non-empty directory and all the files within it.
        - rm -d [dirname] or rmdir [dirname]: Delete an empty directory
      - mv [filename] [new_filename] = Rename the file to a new filename
      - cp -r [dirname] [new_dirname] = Copy dirname to new_dirname

    # Display OS Information
      - uname -a: Show system and kernel
      Linux k8s-node1 5.4.0-1043-raspi #47-Ubuntu SMP PREEMPT Mon Aug 23 09:41:31 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

      ➜ cat /etc/os-release
      NAME="Ubuntu"
      VERSION="20.04.3 LTS (Focal Fossa)"
      ID=ubuntu
      ID_LIKE=debian
      PRETTY_NAME="Ubuntu 20.04.3 LTS"
      VERSION_ID="20.04"
      HOME_URL="https://www.ubuntu.com/"
      SUPPORT_URL="https://help.ubuntu.com/"
      BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
      PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
      VERSION_CODENAME=focal
      UBUNTU_CODENAME=focal

      - lscpu:
      - lsmem:
      - sudo: Allows regular users to run programs with the security privileges of the superusere or root
      - su - chyi: switch user

    # Some More Useful Commands
      - clear: Clears the terminal
      - history: Gives a list of all past commands typed in the current terminal session
      - Ctrl + r: Search history
      - Ctrl + c: Stop current command
      - cat [filename]: Display the file content

  • Package Manager

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
      1. central plcae to install, upgrade,configure, remove software

    # APT - Advanced Package Tool
      > Dependencies are shared
      > Only for specific Linux distributions (package type .deb)
      > Manual Updates
      - apt search <package_name> = Search for a given package
      - apt install <package_name> = Install a given package
      - apt remove <package_name> = Remove installed package
      - apt update = Updates the packages index

    # Repositories:
      > storage location, containing thousands of programs

    # Snap Package Manager:
      > A snap is a bundle of an app and its dependencies
      > Supports universal Linux packages (package type .snap)
      > Automatic Updates
      - Self-contained - dependencies contained in the package

    # Add Repository to official list of repos (add-apt-repository)
    # PPA = Personal Package Archive
      - PPAs are provided by the community

    # Debian Based:   -- APT/APT-GET
      - Ubuntu
      - Debian
      - Mint
    # Red Had Based:  -- Yum
      - RHEL
      - CentOS
      - Fedora

  • Vim Editor

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    # Vim Mode:
      - Command Mode:
        - Navigate, Search, Delete, Undo etc.
        - Whatever you type is interpreted as a command
      - Insert Mode:
    #
    - Press i key = Switch to insert Mode
    - Press esc key = Switch to Command Mode
    - Type dd = Delete entire line
    - Type d10 = Delete next 10 lines
    - Type u = Undo
    - Type $ = Jump to end of the line
    - Type 0 = Jump to start of the line
    - Type /pattern = Search for pattern
    - Type :%s/old/new = Replace old with new throughout the file

  • Users & Permissions

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    # User Accounts:
      - Superuser Account:
      - User Account:
      - Service Account:
        - Don't run services with root user!

      - adduser <username> = Create a new user

    # Groups & Ownership
      - User Level:
      - Group Level:

    # Access Control Files:
      - /etc/passwd:
        - Stores user account information
        - Everyone can read it, but only root user can change the file

        ➜ cat /etc/passwd | grep pi
        pi:x:1000:1000:,,,:/home/pi:/usr/bin/zsh
        USERNAME:PASSWORD:UID:GID:GECOS:HOMEDIR:SHELL
          - PASSWORD(x) means, that encrypted password is stored in /etc/shadow file
          - User ID(UID): Each user has a unique ID. UID 0 is reserved for root
      - /etc/shadow
      - /etc/group

    # File Permissions

    # Linux Commands for Managing Users and their permissions
      - passwd <username> = Change password of a user
      - su - <username> = Login as username (su = switch user)
      - su - = Login as root
      - groupadd <groupname> = Create new group
      - usermod [OPTIONS] <username> = Modify a user account
      - useradd [OPTIONS] <username> = Create a new user
      - groups

    # Different User & Group Commands
      - adduser/addgroup/deluser/delgroup:
        - More User Friendly
      - useradd/groupadd/userdel/groupdel:
        - Low-level utilties

  • Shell Scripting

  • Environmemt Variables

  • Networking

  • SSH - Secure Shell

Version Control

  • GUI Git
    • github
    • gitlab
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      # Git global setup
        $ git config --global user.name ""
        $ git config --global user.email ""

      # Create a new repository
        $ git clone git@gitlab.com:

      # Push an existing folder
        $ cd existing_folder
        $ git init
        $ git remote add origin https://gitlab.com/chyiyaqing/my-project.git
        $ git branch -M main
        $ git push -uf origin main

      # 使用SSH Key
        - GitHub/GitLab can authenticate us when pushing/pulling from repository
        - without providing username and password each time.
  • Git
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    # Git Stages
      - Working Directory (git add -> Staging Area)
      - Staging Area (git commit -> Local Repository)
      - Local Repository

    # Git Command Tools
      - git status : status of local git repo
      - git add:
      - git commit:
      - git log: history of local repository changes
      - git init:
      - git pull:
      - git checkout <branch name> = switch branches
      - git checkout -b <branch name> = creating and switching to new branch
    # Branch
      - Master branch = main branch is created by default, when initializing a git repo
        - best practice: 1 branch per bugfix or feature
        - big feature branches long open, increase the chanceee of merge conflicts
        - only master branch for continuous integration/delivery
        - pipeline is triggered whenever feature/bugfix code is merged into master
        - must be stable and ready for production
      - Develop Branch
        - dev branch: intermediary master
      - Avoiding Merge Commits (rebase)
        - git pull(git fetch + git merge) vs. git pull --rebase (no merge commit, much cleaner project history)
      - Resolving Merge Conflicts
      - Don't track certain files (.gitignore file)
        - .gitignore: to exclude certain folders or files from git to be tracked
        - git rm -r --cached : Stop tracking a file
      - Save work-in-progress changes (stash)
        - git stash (save un-committed (work-in-progress) changes, unfinished changes, which don't want to commit yet)
        - git stash pop (get the changes back)
      - Going back in history
        - git checkout <commit hash> (go back to a specific project version)
        - Each commit: (unique commit hash, commit message, author + date)
      - Undoing and changing commits
        - git reset --hard <HEAD~1> (removed old commit)
        - git reset --soft (keep the changes in your working directory)
        - git revert <commit hash> (creates a new commit to revert the old commits changesj)
        - git commit --amend (changing the last commit)
      - Merging Branches

    # Merge Requests:
      - Best practice: Other developer reviews code changes before merging
    # Git 名词解释
      - origin = your remote git project/repo

    # Git for DevOps
      - Infrastructure as Code
        - Many Kubernetes Configuration Files (Yaml) (Deployment to Kubernetes)
        - Terraform and Ansible Configuration Files
        - Bash and Python Python Scripts
        - tracked - history of changes
        - securely stored in one place
      - CI/CD Pipeline and Build Automation
        - checkout code, test and build application, etc.
        - need integration for the build automation tool with application git repository
  • Databases
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    > Database are used to persist data!
    - DB endpoint & credentials should not be hardcode

    # DevOps Engineer
      - how to configure DB
      - hot to setup DB
      - how to manage DB

    # Database Types
      - Key-Value Database
        - Redis
        - Memcached
        - etcd from Kubernetes
        # Best for:
          - Caching
          - Message Queue
      - Wide Column Database (schema-less, scalable, no joins)
        - Cassandra
        - HBase
        # Best for:
          - TIME-Series
          - Iot-Records
          - Historical-records
      - Document Databases (schema-less)
        - MongoDB
        - DynamoDB
        - CouchDB
        # Best for:
          - Mobile Apps
          - Game Apps
          - Cms
          - Most Apps
      - Relational Databases (schema, structured query language )
        - MySQL
        - Postgresql
        # Best for:
          - ACID (Atomicity, Consistency, Isolation, Durability)
      - Graph Database (Nodes and Edges)
        - Neo4j
        - Dgraph
        # Best for:
          - Graphs
          - Patterns
      - Search Database
        - ElasticSearch
        - Solr

Build Tools & Package Manager

  • artifact
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    # Building the code:
      - compiling
      - Compress

    # Aritfact repositoy
      > Keep artifact in storage
      > to deploy it multiple times, have backup etc.
      - Nexus
      - JFrog

    # What kind of file is the artifact?
      - Java artifact (JAR or WAR file includes whole code plus dependencies)

    # Install Java and Build Tools
      - Install Java
      - Install Maven (mvn package)
      - Install Gradle (./gradlew build)
      - Install Node + npm

    # Build Aritifact
      - 1. Using a Build Tool (Install dependencies; Compile and compress your code)
        Java: (JAR or WAR file)
          - Maven (Using XML)
            - mvn install
            - mvn package (target/xxx-.jar)
          - Gradle (Using Groovy)
            - ./gradlew build (build/libs/xxx.jar)
            - Note: Gradle-Java Version Compatibility (Change Java version less than 16)

    # Managing Dependencies
      > Dependencies file = managing the dependencies for a project
      - maven (pom.xml)
      - gradle (build.gradle)

      - Local Dependency Repository(~/.m2/repository)

    # Run the application
      - JAR (java -jar <name of jar file>)

    # Build JS Applications
      - no special artifact type (ZIP/TAR file)
      - npm and yarn
      - package.json file for dependencies
      - package managers and NOT build tools
      - install dependencies, but not used for transpiling JS code
      - Command Lint Tool - npm
        - npm install = install dependencies
        - npm start - start the application
        - npm stop - stop the application
        - npm test - run the tests
        - npm publish - publish the artifact
        - [ ] npm pack - package
      - zip/tar file include?
        - application code, but not the dependencies
        - must install the dependencies first
        - unpack zip/tar
        - run the APP (copy artifact & package.json file)

    # Build Tool - Webpack
      > transpiles(转译)|minifies(缩小)|bundles(捆绑)|compresses(压缩) the code

    # What are Build Tools for other programming language?
      - Python (pip)

    # Pattern in all these tools
      - dependecyfile
        - maven (pom.xml)
        - gradle (build.gradle)
        - npm (package.json)
      - repository for dependencies
      - command line tool
      - package managers

    # Build Tools for DevOps Engineers
      - Build Docker Image => Push to Repo => Run on Server
      - Install Dependencies => run tests => build/bundle app => push to repo

    SDK = Software Development Kit

Cloud & Infrastructure as a Service

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# What is Infrastructure as a Service?
  - Buys own server
    > manage own server and infrastructure
    > if something breaks, need to fix it
  - Delegate Infrastructure Management
    > Move your physical infrastructure to cloud
    > Just rent (租赁) the servers

# Infrastructure as a Service Providers?
  - Google Cloud
  - aws
  - DigitalOcean
  - Microsoft Azure

> Inbound = Incoming Requests
> Outbound = outgoing requests from server
  • Create a Linux User (Not use Root)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    # Security Best practice
      - Create separate User for every application
      - Give it only the permission it needs to run that App
      - Don't work with the Root user

      > create username
      $ adduser <username>

      > Add user to "sudo" group
      $ usermod -aG sudo <username>

      > switch to username
      $ su - <username>

      $ = standard Linux User
      # = Root User
  • Nexus: Artifact Repository
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    > Nexus is one of the most popular
    > upload and store different built artifacts
    > retrieve(download) artifacts later
    > central storage
    > host own repositories
    > Proxy Repository
    > metadata tagging (labelling and tagging artifacts)
    > cleanup policies
    > search functionality
    > user token support for system user authentication

    # What is an Artifact Repository?
      - Artifacts = Apps built into a **single file**
      - Different Artifact formats (JAR, WAR, ZIP, TAR)
      - Artifact repository = storage of those artifacts

    # What is an Artifact Repository Manager?
      - Repository Formats?
        - APT
        - Composer
        - Conan
        - CPAN
        - Docker
        - ELPA
        - Git LFS
        - Go
        - Helm
        - Maven
        - npm
        - NuGet
        - P2
        - PyPI
        - R
        - Raw
        - RubyGems
        - Yum

    Jenkins --> push --> Nexus <-- pull <-- Server

    # Nexus Install
      > Download Nexus tar file
      $ wget https://download.sonatype.com/nexus/3/latest-unix.tar.gz

      > Untar file
      $ tar -xvf latest-unix.tar.gz

      chyi in ~/Downloads at k8s-master
      ➜ ll
      total 196M
      -rw-r--r--  1 chyi users 196M Oct 12 19:58 latest-unix.tar.gz
      drwxr-xr-x 10 chyi users 4.0K Oct 17 15:08 nexus-3.35.0-02      -- Nexus folder: (contains runtime and application of Nexus)
      drwxr-xr-x  3 chyi users 4.0K Oct 17 15:08 sonatype-work        -- Sonatype-work: contains own config for Nexus and data
        - subdirectories depending on your Nexus configuration
        - IP address that accessed Nexus
        - Logs of Nexus App
        - Your uploaded files and metadata

    # Staring Nexus
      - Services should not run with Root User Permissions
      - Best practice: Create own User for Service (e.g. Nexus - Only the permission for that specific Service)

      > Create User
      $ adduser nexus

      > change folder owner
      > Nexus User execute nexus executable
      > Nexus read/write sonatype-work folder
      $ chown -R nexus:nexus nexus-3
      $ chown -R nexus:nexus sonatype-work

      > change run user
      $ sudo vim nexus-3/bin/nexus.rc
        run_as_user="nexus"

      > start nexus
      nexus@chyiyaqing-PowerEdge-R720:/opt$ ./nexus-3.35.0-02/bin/nexus start
      Starting nexus

      > check nexus status
      nexus@chyiyaqing-PowerEdge-R720:/opt$ netstat -lnpt | grep 8081
      (Not all processes could be identified, non-owned process info
       will not be shown, you would have to be root to see it all.)
      tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      2732557/java

      > Access Nexus from Browser
      $ curl http://localhost:8081

      > Create new Repository
      > Hosted Repository
      > Group Repository

    # Publish artifact to Nexus Repository
      > Upload Jar File to existing hosted repository on Nexus
      > Maven/Gradle command for pushing to remote repository
      > Configure both tools to connect to Nexus (Nexus Repo URL + Credentials)
      > Nexus User with permission to upload

      - Create Nexus User
      - Gradle Project Configure with Nexus

    # Upload Jar File to Nexus
      - Upload Jar File to existing hosted repository on Nexus
      - Maven/Gradle command for pushing to remote repository
      - Configure both tools to connect to Nexus(Nexus Repo URL + Credentials)
      - Nexus User with permission to upload

    # Nexus API
      - Query Nexus Repository for different information
        - Use a tool like **curl** or **wget** to execute http request
        # List all repositories
        $ curl -u user:pwd -X GET 'http://IP:HOST/service/rest/v1/repositories'
          [ {
            "name" : "nuget-group",
            "format" : "nuget",
            "type" : "group",
            "url" : "http://172.30.1.12:8081/repository/nuget-group",
            "attributes" : { }
          }, {
            "name" : "maven-snapshots",
            "format" : "maven2",
            "type" : "hosted",
            "url" : "http://172.30.1.12:8081/repository/maven-snapshots",
            "attributes" : { }
          }, {
            "name" : "maven-central",
            "format" : "maven2",
            "type" : "proxy",
            "url" : "http://172.30.1.12:8081/repository/maven-central",
            "attributes" : {
              "proxy" : {
                "remoteUrl" : "https://repo1.maven.org/maven2/"
              }
            }
          }, {
            "name" : "nuget.org-proxy",
            "format" : "nuget",
            "type" : "proxy",
            "url" : "http://172.30.1.12:8081/repository/nuget.org-proxy",
            "attributes" : {
              "proxy" : {
                "remoteUrl" : "https://api.nuget.org/v3/index.json"
              }
            }
          }, {
            "name" : "maven-releases",
            "format" : "maven2",
            "type" : "hosted",
            "url" : "http://172.30.1.12:8081/repository/maven-releases",
            "attributes" : { }
          }, {
            "name" : "nuget-hosted",
            "format" : "nuget",
            "type" : "hosted",
            "url" : "http://172.30.1.12:8081/repository/nuget-hosted",
            "attributes" : { }
          }, {
            "name" : "maven-public",
            "format" : "maven2",
            "type" : "group",
            "url" : "http://172.30.1.12:8081/repository/maven-public",
            "attributes" : { }
          } ]%

        # List components
        ➜ curl -u admin:xx -X GET 'http://172.30.1.12:8081/service/rest/v1/components?repository=maven-snapshots'

    # Blob Stores
      - Nexus Storage to store all the uploaded files
      - Storage of binary files
      - Local Storage or Cloud Storage
      - Type field = Storage Backend
        - File: file system-based storage
        - s3: cloud-based storage
    • Blob Stores: Configure local and cloud blob storage
      Nexus Blob Stores
      1
      2
      3
      4
      5
      6
      7
      8
      9
      > Blob store can't be modified
      > Blob store used by a repository can't be deleted!
      # Type:
      # State: state of the blob store
        - started: indicates it's running as expected
        - failed: indicates a configuration issue
      # Blob Count: number of blobs currently stored
      # Total Size:
      # Available space:
    • Component vs Asset
      1
      2
      3
      4
      5
      6
      # Component:
        - abstract
        - what we are uploading
      # Asset:
        - actual physical packages/files
        - 1 component = 1 or more assets
  • Jenkins: Build Automation
  • LDAP

Infrastructure Privisioning

  • Containers with Docker
    • What is a Container?
      1
      2
      3
      4
      5
      6
      7
      > A way to package application with all the necessary dependencies and configuration
      > Portable artifact, easily shared and moved around

      # Container Technologies
        - Docker
        - Containerd
        - CRI-O
  • Container Vs Image
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    # Docker Image:
      > the actual package
      - artifact, that can be moved around

    # Docker Container
      > actually start the application
      > container environment is created
      > is a running environment for Image
      - virtual File system
      - environment configs
      - application images
  • Docker vs Virtual Machine
    1
    2
    3
    > Operating System have 2 Layers:
      - OS Kernel
      - Application [Docker]
  • Docker Architecture & Components
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    # Docker Engine:
      - Server:
        - Container Runtime:
          - pull images
          - managing images & containers
        - Volumes:
          - persisting data
        - Network:
          - configuring network for container communication
        - build images:
          - build own Docker images
      - API:
        - interacting with Docker Server
      - CLI:
        - Command Line Interface: client to execute docker commands
  • Main Docker commands
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    # Images

    # Container

    # Container Port vs. Host Port

    # Debugging Docker Containers

    $ docker exec
      - docker exec -it container_id /bin/bash
    $ docker pull
    $ docker run
      -d : daemon
      docker run -d -p6000:6379 --name redis redis:latest
    $ docker exec -it
    $ docker logs
    $ docker images
    $ docker ps = list running containers
      -a : lists running and stopped container
    $ docker run = starts new container with a command
    $ docker start = starts the container
    $ docker stop = stops the container
    $ docker network ls
  • Demo Overview
    Demo Workflow with Docker
    • Develop with Docker
      1
      2
      3
      4
      5
      # Demo project
        - JS and Nodejs application
          - localhost:3000/my-app
        - MongoDB Docker Container
          - localhost:3001/db/my-db
  • Run multiple containers - Docker Compose
  • Build Docker Image - Dockerfile
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    # What is a Dockerfile?
      - Blueprint for building images

    # FROM : Start by basing it on another image
      FROM node

    # ENV var: Optionally define environment variables
      ENV MONGO_DB_USERNAME=admin \
          MONGO_DB_PWD=password

    # RUN : execute any linux command
      RUN mkdir -p /home/app

    # COPY : execute on the HOST machine
      COPY . /home/app

    # CMD : entrypoint command
      CMD ["node", "server.js"]

    $ docker build -t my-app:version-1 .
    $ docker run my-app:version-1
  • Push to private Docker Repository
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    # Docker private repository
      - Harbar:

    # build & tag an images
      docker tag chyiyaqing/alpine_sl harbor.chyidl.com/chyiyaqing/alpine_sl

    # docker login
      ➜ docker login -u admin -p macintosh harbor.chyidl.com
      WARNING! Using --password via the CLI is insecure. Use --password-stdin.
      WARNING! Your password will be stored unencrypted in /home/chyiyaqing/.docker/config.json.
      Configure a credential helper to remove this warning. See
      https://docs.docker.com/engine/reference/commandline/login/#credentials-store

      Login Succeeded

    # docker push
      ➜ docker push harbor.chyidl.com/chyiyaqing/alpine_sl
      Using default tag: latest
      The push refers to repository [harbor.chyidl.com/chyiyaqing/alpine_sl]
      102b73ff0977: Pushed
      e2eb06d8af82: Pushed
      latest: digest: sha256:500e9b9536a4fec460c487867d3cd7084da934ba37151aa122f4be438cc06d57 size: 739
  • Persist data in Docker - Volumes
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    # When do we need Docker Volumes?
      > For data persistence

    # What is Docker Volumes

    # 3 Volumes Types
      - Host Volumes (you decide where on the host file system the reference is made)
        - -v /home/mount/data:/var/lib/mysql/data

      - Anonymous Volumes (automatically created by Docker)
        - -v /var/lib/mysql/data

      - Named Volumes
        - -v name:/var/lib/mysql/data

    # Docker Volume Locations
      - Windows: C:\ProgramData\docker\volumes
      - Linux: /var/lib/docker/volumes
      - MacOS: /var/lib/docker/volumes

    # Docker for Mac
      > creates a Linux virtual machine and stores all the Docker data here
  • Push/Pull Nexus Repository
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    # Docker login to Nexus Docker Repo
      chyiyaqing in ~ at chyiyaqing-PowerEdge-R720
      ➜ docker login 172.30.1.12:8083
      Username: chyi
      Password:
      WARNING! Your password will be stored unencrypted in /home/chyiyaqing/.docker/config.json.
      Configure a credential helper to remove this warning. See
      https://docs.docker.com/engine/reference/commandline/login/#credentials-store

      Login Succeeded

    # Push Images to Nexus Repo
      ➜ docker tag ubuntu:latest 172.30.1.12:8083/ubuntu:latest

      chyiyaqing in ~ at chyiyaqing-PowerEdge-R720
      ➜ docker push 172.30.1.12:8083/ubuntu:latest
      The push refers to repository [172.30.1.12:8083/ubuntu]
      9f54eef41275: Pushed
      latest: digest: sha256:7cc0576c7c0ec2384de5cbf245f41567e922aab1b075f3e8ad565f508032df17 size: 529

    # Fetch Docker Image from Nexus
      chyiyaqing in ~ at chyiyaqing-PowerEdge-R720 took 7s
      ➜ curl -u chyi:macintosh -X GET 'http://172.30.1.12:8081/service/rest/v1/components?repository=docker-hosted'
      {
        "items" : [ {
          "id" : "ZG9ja2VyLWhvc3RlZDo3MWIyYzQyZWUyMjFkZTJlNzZlYTY4YjNmMGY1MTQ4Mw",
          "repository" : "docker-hosted",
          "format" : "docker",
          "group" : null,
          "name" : "ubuntu",
          "version" : "latest",
          "assets" : [ {
            "downloadUrl" : "http://172.30.1.12:8081/repository/docker-hosted/v2/ubuntu/manifests/latest",
            "path" : "v2/ubuntu/manifests/latest",
            "id" : "ZG9ja2VyLWhvc3RlZDoyNjlmOTEwOWZiZmIyMzlmNTc1OTU2MGQ1NmZjZTQ5Nw",
            "repository" : "docker-hosted",
            "format" : "docker",
            "checksum" : {
              "sha1" : "064d46bcae0d9c485376fbeabeed1f3e4f1c8679",
              "sha256" : "7cc0576c7c0ec2384de5cbf245f41567e922aab1b075f3e8ad565f508032df17"
            },
            "contentType" : "application/vnd.docker.distribution.manifest.v2+json",
            "lastModified" : "2021-11-13T06:33:59.622+00:00",
            "blobCreated" : "2021-11-13T06:33:59.622+00:00",
            "lastDownloaded" : null
          } ]
        } ],
        "continuationToken" : null
      }%
  • Run Nexus as Docker Container
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    # Persistent Data
      $ docker volume create --name nexus-data
      $ docker run -d -p 8081:8081 --name nexus -v nexu-data:/nexus-data sonatype/nexus3
        -d = detached, run in background
        -p 8081:8081 = make accessible at 8081
        --name = container name
        -v = volume(named volume here)

      chyi in ~ at k8s-master
      ➜ netstat -lnpt | grep 8081
      (Not all processes could be identified, non-owned process info
       will not be shown, you would have to be root to see it all.)
      tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      -

      chyi in ~ at k8s-master
      ➜ docker ps
      CONTAINER ID   IMAGE             COMMAND                  CREATED          STATUS          PORTS                    NAMES
      8630fd5d8faa   sonatype/nexus3   "sh -c ${SONATYPE_DI…"   26 seconds ago   Up 23 seconds   0.0.0.0:8081->8081/tcp   nexus

    chyi in ~ at k8s-master
    ➜ docker volume ls
    DRIVER    VOLUME NAME
    local     nexus-data

    chyi in ~ at k8s-master
    ➜ docker inspect nexus-data
    [
        {
            "CreatedAt": "2021-11-13T15:09:41+08:00",
            "Driver": "local",
            "Labels": {},
            "Mountpoint": "/mnt/INNERDISK/docker/volumes/nexus-data/_data",
            "Name": "nexus-data",
            "Options": {},
            "Scope": "local"
        }
    ]

    chyi in ~ at k8s-master
    ➜ ls /mnt/INNERDISK/docker/volumes/nexus-data/_data
    blobs  db             etc                instances  karaf.pid  lock  orient  restore-from-backup
    cache  elasticsearch  generated-bundles  javaprefs  keystores  log   port    tmp

Build Automation & CI/CD with Jenkins

  • Introduction to Build Automation

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    # What is Build Automation?
      Test Code -> Build Application -> Push to Repository -> Deploy to Server

    # What is Jenkins?

    # What can you do with Jenkins?
      Run tests
      Build artifacts
      Publish artifacts
      Deploy artifacts
      Send notifactions

    # Integration with other technologies
      Docker
      Repositories
      Build Tools
      Deployment Servers

    # What do you need to configure?
      Run test -- Build Tools need to be available
      Build Application -- Build Tools or Docker available
      Publish Docker Image -- Store Credentials in Jenkins

  • Deploy Jenkins on DigitalOcean server

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    # Install Jenkins directly on OS
    # Run Jenkins as Docker container
      $ docker run -p 8080:8080 -p 50000:50000 -d -v jenkins_home:/var/jenkins_home jenkins/jenkins:lts

      chyiyaqing in ~ at chyiyaqing-PowerEdge-R720 took 15s
      ➜ docker ps | grep jenkins
      e01cdfdb64f1   jenkins/jenkins:lts                                             "/sbin/tini -- /usr/…"   14 seconds ago       Up 12 seconds                      0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:50000->50000/tcp, :::50000->50000/tcp                                    brave_mcnulty

      chyiyaqing in ~ at chyiyaqing-PowerEdge-R720
      ➜ docker volume inspect jenkins_home
      [
          {
              "CreatedAt": "2021-11-14T02:04:28+08:00",
              "Driver": "local",
              "Labels": null,
              "Mountpoint": "/var/lib/docker/volumes/jenkins_home/_data",
              "Name": "jenkins_home",
              "Options": null,
              "Scope": "local"
          }
      ]

  • Roles of Jenkins

    1
    2
    3
    4
    5
    6
    7
    8
    # Jenkins Administrator (Operations or DevOps teams)
      - administers and manages Jenkins
      - sets up Jenkins cluster
      - installs plugins
      - backup Jenkins data

    # Jenkins User (Developer or DevOps teams)
      - creating the actual jobs to run workflows

  • Install and use Build Tools in Jenkins

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    # Create Job to automate your apps workflow
      - Java App (Maven)
        - Java App with Maven Build Tool
        - Run Tests
        - Build Jar File
      - JavaScript App (npm)
        - Node App
        - Run Tests
        - Package and push to Repo

    # 2 ways to install and configure those tools
      - Jenkins Plugins
        - just install plugin (via UI) for your tool

      - Install Tools directly on Server
        - enter in remote server and install
        - inside the Docker container, when Jenkins runs as container

    # Install npm and Node in Jenkins container
      chyiyaqing in ~ at chyiyaqing-PowerEdge-R720 took 19s
      ➜ docker exec -u 0 -it e01cdfdb64f1 bash (root user)
      root@e01cdfdb64f1:/# cat /etc/issue
      Debian GNU/Linux 11 \n \l

      root@e01cdfdb64f1:/# sed -i 's/deb.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list (替换国内源)
      root@e01cdfdb64f1:/# curl -sL https://deb.nodesource.com/setup_16.x -o nodesource_setup.sh
      root@e01cdfdb64f1:/# ls
      bin  boot  dev	etc  home  lib	lib64  media  mnt  nodesource_setup.sh	opt  proc  root  run  sbin  srv  sys  tmp  usr	var
      root@e01cdfdb64f1:/# bash nodesource_setup.sh

      ## Installing the NodeSource Node.js 16.x repo...

  • Create Freestyle Job

    1
    2
    3
    4
    5
    6
    7
    - Plugins
    - Credentials
    - Configure Git Repository

    - Plugin Configuration
      1. Install directly on Server: more flexible
      2. Plugin: limited to provided input fields

  • Docker in Jenkins

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    chyiyaqing in ~/chyi at chyiyaqing-PowerEdge-R720
    ➜ docker run -p 8080:8080 -p 50000:50000 -d \
    ➜ -v jenkins_home:/var/jenkins_home \
    ➜ -v /var/run/docker.sock:/var/run/docker.sock \
    ➜ -v $(which docker):/usr/bin/docker jenkins/jenkins:lts
    9b9acf0e8c13519d3bc5a96b8e7b2d251374e7112a796d2b9cc55e8ee53361a1

    jenkins@9b9acf0e8c13:/$ docker pull redis
    Using default tag: latest
    Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/create?fromImage=redis&tag=latest": dial unix /var/run/docker.sock: connect: permission denied
    jenkins@9b9acf0e8c13:/$

    # Build Docker Image

    # Push Image to DockerHub
      - Create DockerHub account
      - Configure that credentials in Jenkins

    # Push Docker Image to Nexus Repository

    # Freestyle to Pipeline Job
      - Freestyle Job with multiple Steps
        - Build Java App -> Run Tests -> Build Image -> Push to private Repo

    # Pipeline Job
      - Pipeline script
        - Script is written in Groovy
        - Programming Language similar to Java

      - Required Fields of Jenkinsfile
        # pipeline must be top-level
        pipeline {
            // agent where to execute
            agent any

            // Access Build Tools for your projects
            tools {
              // Only 3 Build tools available: gradle, maven and JDK!
            }
            environment {
              NEW_VERSION = "1.0"
              SERVER_CREDENTIALS = credential('server-credentials')
            }
            parameters {
              string(name: 'VERSION', defaultValue: '', description: 'version to deploy on prod')
              choice(name: 'VERSION', choices: ['1.1.0', '1.2.0'], description: '')
            }

            // stages - where the work happens
            stages {
                stage("build") {
                  when {
                    expression {
                      expression {
                        BRANCH_NAME == "dev" || CODE_CHANGES == true
                      }
                    }
                  }
                    steps {
                      echo 'building the application...'
                      echo "building version ${NEW_VERSION}"
                    }
                }

                stage("test") {
                  when {
                    expression {
                      expression {
                        BRANCH_NAME == "dev" || BRANCH_NAME == "master"
                      }
                    }
                  }
                  steps {
                      echo 'building the application...'
                  }
                }

                stage("deploy") {
                    steps {
                      echo 'building the application...'
                      echo "deploying with $(SERVER_CREDENTIALS)"
                    }
                }
            }

            // Execute some logic AFTER all stages executed
            post {

              // Conditions
              always {

              }

              success {

              }

              failure {

              }
            }
        }

    # Jenkins Variables
      $ http://172.30.1.12:8080/env-vars.html/

  • Build scripted Pipeline

  • Henkinsfile in detail

  • Configure automated versioning

  • Jenkins Shared Library

Programming

Containers

Container Orchestration

Configuration Management

Monitoring

Command Line Commands

  • scp = secure copy

    scp

  • netstat

    netstat -lpnt

专业名词

  • LTS = Long Term Support

Wrap Up: 总结

  • Pipeline
    Pipeline 开发流程

参考资料