Episode 98 – 99 – Vagrant and Terraform

Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the “works on my machine” excuse a relic of the past.

Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can automatically install and configure software on the virtual machine.

For Developers

If you are a developer, Vagrant will isolate dependencies and their configuration within a single disposable, consistent environment, without sacrificing any of the tools you are used to working with (editors, browsers, debuggers, etc.). Once you or someone else creates a single Vagrantfile, you just need to vagrant up and everything is installed and configured for you to work. Other members of your team create their development environments from the same configuration, so whether you are working on Linux, Mac OS X, or Windows, all your team members are running code in the same environment, against the same dependencies, all configured the same way. Say goodbye to “works on my machine” bugs.

»For Operators

If you are an operations engineer or DevOps engineer, Vagrant gives you a disposable environment and consistent workflow for developing and testing infrastructure management scripts. You can quickly test things like shell scripts, Chef cookbooks, Puppet modules, and more using local virtualization such as VirtualBox or VMware. Then, with the same configuration, you can test these scripts on remote clouds such as AWS or RackSpace with the same workflow. Ditch your custom scripts to recycle EC2 instances, stop juggling SSH prompts to various machines, and start using Vagrant to bring sanity to your life.

»For Designers

If you are a designer, Vagrant will automatically set everything up that is required for that web app in order for you to focus on doing what you do best: design. Once a developer configures Vagrant, you do not need to worry about how to get that app running ever again. No more bothering other developers to help you fix your environment so you can test designs. Just check out the code, vagrant up, and start designing.

»For Everyone

Vagrant is designed for everyone as the easiest and fastest way to create a virtualized environment!

»Vagrant Vs Other Tools

Vagrant is not the only tool to manage virtual machines and development environments. This section compares Vagrant to these other software choices.

Vagrant vs. CLI Tools

Virtualization software like VirtualBox and VMware come with command line utilities for managing the lifecycle of machines on their platform. Many people make use of these utilities to write their own automation. Vagrant actually uses many of these utilities internally.

The difference between these CLI tools and Vagrant is that Vagrant builds on top of these utilities in a number of ways while still providing a consistent workflow. Vagrant supports multiple synced folder types, multiple provisioners to setup the machine, automatic SSH setup, creating HTTP tunnels into your development environment, and more. All of these can be configured using a single simple configuration file.

Vagrant still has a number of improvements over manual scripting even if you ignore all the higher-level features Vagrant provides. The command-line utilities provided by virtualization software often change each version or have subtle bugs with workarounds. Vagrant automatically detects the version, uses the correct flags, and can work around known issues. So if you’re using one version of VirtualBox and a co-worker is using a different version, Vagrant will still work consistently.

For highly-specific workflows that don’t change often, it can still be beneficial to maintain custom scripts. Vagrant is targeted at building development environments but some advanced users still use the CLI tools underneath to do other manual things.

Vagrant vs. Docker

Vagrant is a tool focused on providing a consistent development environment workflow across multiple operating systems. Docker is a container management that can consistently run software as long as a containerization system exists.

Containers are generally more lightweight than virtual machines, so starting and stopping containers is extremely fast. Docker uses the native containerization functionality on macOS, Linux, and Windows.

Currently, Docker lacks support for certain operating systems (such as BSD). If your target deployment is one of these operating systems, Docker will not provide the same production parity as a tool like Vagrant. Vagrant will allow you to run a Windows development environment on Mac or Linux, as well.

For microservice heavy environments, Docker can be attractive because you can easily start a single Docker VM and start many containers above that very quickly. This is a good use case for Docker. Vagrant can do this as well with the Docker provider. A primary benefit for Vagrant is a consistent workflow but there are many cases where a pure-Docker workflow does make sense.

Both Vagrant and Docker have a vast library of community-contributed “images” or “boxes” to choose from.

How to get started

You can follow Vagrant’s Getting Started Guide from the below link:
https://learn.hashicorp.com/collections/vagrant/getting-started

Hashicorp Vagrant Cloud


https://app.vagrantup.com/boxes/search

You can find prebuilt boxes for all scenarios from all operating systems from many different providers (vmware, virtuabox, cloud, etc.)

Screenshot of Vagrant Cloud

What’s in a Base Box?

A base box typically consists of only a bare minimum set of software for Vagrant to function. As an example, a Linux box may contain only the following:

  • Package manager
  • SSH
  • SSH user so Vagrant can connect
  • Perhaps Chef, Puppet, etc. but not strictly required.

In addition to this, each provider may require additional software. For example, if you are making a base box for VirtualBox, you will want to include the VirtualBox guest additions so that shared folders work properly. But if you are making an AWS base box, this is not required.

»Creating a Base Box
( Explained with Virtualbox)

Creating a Base Box

There are a special category of boxes known as “base boxes.” These boxes contain the bare minimum required for Vagrant to function, are generally not made by repackaging an existing Vagrant environment (hence the “base” in the “base box”).

For example, the Ubuntu boxes provided by the Vagrant project (such as “bionic64”) are base boxes. They were created from a minimal Ubuntu install from an ISO, rather than repackaging an existing environment.

Base boxes are extremely useful for having a clean slate starting point from which to build future development environments. The Vagrant project hopes in the future to be able to provide base boxes for many more operating systems. Until then, this page documents how you can create your own base box.

Creating a base box is actually provider-specific. This means that depending on if you are using VirtualBox, VMware, AWS, etc. the process for creating a base box is different. 

As with every Vagrant provider, the Vagrant VirtualBox provider has a custom box format that affects how base boxes are made.

Additionally, it is helpful to understand the basics of the box file format.

Packer and Vagrant Cloud

HashiCorp strongly recommends using Packer to create reproducible builds for your base boxes for CI/CD scenarios as well as automating the builds.

I will not discuss Packer in this episode.

You can read more about automating Vagrant box creation with Packer in the Packer documentation.

»Disk Space

When creating a base box, make sure the user will have enough disk space to do interesting things, without being annoying. For example, in VirtualBox, you should create a dynamically resizing drive with a large maximum size. This causes the actual footprint of the drive to be small initially, but to dynamically grow towards the max size as disk space is needed, providing the most flexibility for the end user.

If you are creating an AWS base box, do not force the AMI to allocate terabytes of EBS storage, for example, since the user can do that on their own. But you should default to mounting ephemeral drives, because they’re free and provide a lot of disk space.

»Memory

Like disk space, finding the right balance of the default amount of memory is important. For most providers, the user can modify the memory with the Vagrantfile, so do not use too much by default. It would be a poor user experience (and mildly shocking) if a vagrant up from a base box instantly required many gigabytes of RAM. Instead, choose a value such as 512MB, which is usually enough to play around and do interesting things with a Vagrant machine, but can easily be increased when needed.

»Peripherals (Audio, USB, etc.)

Disable any non-necessary hardware in a base box such as audio and USB controllers. These are generally unnecessary for Vagrant usage and, again, can be easily added via the Vagrantfile in most cases.

»Default User Settings

Just about every aspect of Vagrant can be modified. However, Vagrant does expect some defaults which will cause your base box to “just work” out of the box. You should create these as defaults if you intend to publicly distribute your box.

If you are creating a base box for private use, you should try not to follow these, as they open up your base box to security risks (known users, passwords, private keys, etc.).

»“vagrant” User

By default, Vagrant expects a “vagrant” user to SSH into the machine as. This user should be setup with the insecure keypair that Vagrant uses as a default to attempt to SSH. Also, even though Vagrant uses key-based authentication by default, it is a general convention to set the password for the “vagrant” user to “vagrant”. This lets people login as that user manually if they need to.

To configure SSH access with the insecure keypair, place the public key into the ~/.ssh/authorized_keys file for the “vagrant” user. Note that OpenSSH is very picky about file permissions. Therefore, make sure that ~/.ssh has 0700 permissions and the authorized keys file has 0600 permissions.

When Vagrant boots a box and detects the insecure keypair, it will automatically replace it with a randomly generated keypair for additional security while the box is running.

»Root Password: “vagrant”

Vagrant does not actually use or expect any root password. However, having a generally well known root password makes it easier for the general public to modify the machine if needed.

Publicly available base boxes usually use a root password of “vagrant” to keep things easy.

»Password-less Sudo

This is important!. Many aspects of Vagrant expect the default SSH user to have passwordless sudo configured. This lets Vagrant configure networks, mount synced folders, install software, and more.

To begin, some minimal installations of operating systems do not even include sudo by default. Verify that you install sudo in some way.

After installing sudo, configure it (usually using visudo) to allow passwordless sudo for the “vagrant” user. This can be done with the following line at the end of the configuration file:

vagrant ALL=(ALL) NOPASSWD: AL

Additionally, Vagrant does not use a pty or tty by default when connected via SSH. You will need to make sure there is no line that has requiretty in it. Remove that if it exists. This allows sudo to work properly without a tty. Note that you can configure Vagrant to request a pty, which lets you keep this configuration. But Vagrant by default does not do this.

»SSH Tweaks

In order to keep SSH speedy even when your machine or the Vagrant machine is not connected to the internet, set the UseDNS configuration to no in the SSH server configuration.

This avoids a reverse DNS lookup on the connecting SSH client which can take many seconds.

»Windows Boxes

Supported Windows guest operating systems:

  • Windows 7
  • Windows 8
  • Windows 10 / 11
  • Windows Server 2008
  • Windows Server 2008 R2
  • Windows Server 2012
  • Windows Server 2012 R2
  • Windows 2016
  • Windows 2019 / 2022

Windows Server 2003 and Windows XP are not supported, but if you are a die hard XP fan this link may help you.

»Base Windows Configuration

  • Turn off UAC
  • Disable complex passwords
  • Disable “Shutdown Tracker”
  • Disable “Server Manager” starting at login (for non-Core)

In addition to disabling UAC in the control panel, you also must disable UAC in the registry. This may vary from Windows version to Windows version, but Windows 8/8.1 use the command below. This will allow some things like automated Puppet installs to work within Vagrant Windows base boxes.

reg add HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /d 0 /t REG_DWORD /f /reg:64

»Base WinRM Configuration

To enable and configure WinRM you will need to set the WinRM service to auto-start and allow unencrypted basic auth (obviously this is not secure). Run the following commands from a regular Windows command prompt:

winrm quickconfig -qwinrm set winrm/config/winrs @{MaxMemoryPerShellMB="512"}winrm set winrm/config @{MaxTimeoutms="1800000"}winrm set winrm/config/service @{AllowUnencrypted="true"}winrm set winrm/config/service/auth @{Basic="true"}sc config WinRM start= auto

»Additional WinRM 1.1 Configuration

These additional configuration steps are specific to Windows Server 2008 (WinRM 1.1). For Windows Server 2008 R2, Windows 7 and later versions of Windows you can ignore this section.

  1. Ensure the Windows PowerShell feature is installed
  2. Change the WinRM port to 5985 or upgrade to WinRM 2.0

The following commands will change the WinRM 1.1 port to what’s expected by Vagrant:

netsh firewall add portopening TCP 5985 "Port 5985"winrm set winrm/config/listener?Address=*+Transport=HTTP @{Port="5985"}

»Optional WinSSH Configuration

When using the WinSSH communicator, you may run into an issue where a PowerShell command can’t display a progress bar. A typical error message might look like:

Win32 internal error "Access is denied" 0x5 occurred while reading the console output buffer.

In order to prevent this, we recommend setting $ProgressPreference = "SilentlyContinue" in your box’s PowerShell profile:

if (!(Test-Path -Path $PROFILE)) {  New-Item -ItemType File -Path $PROFILE -Force}
Add-Content $PROFILE '$ProgressPreference = "SilentlyContinue"'

»Other Software , »Additional Software

At this point, you have all the common software you absolutely need for your base box to work with Vagrant. However, there is some additional software you can install if you wish.

While we plan on it in the future, Vagrant still does not install Chef or Puppet automatically when using those provisioners. Users can use a shell provisioner to do this, but if you want Chef/Puppet to just work out of the box, you will have to install them in the base box.

Installing this is outside the scope of this page, but should be fairly straightforward.

In addition to this, feel free to install and configure any other software you want available by default for this base box.

In addition to the software that should be installed based on the general guide to creating base boxes, VirtualBox base boxes require some additional software.

»VirtualBox Guest Additions

VirtualBox Guest Additions must be installed so that things such as shared folders can function. Installing guest additions also usually improves performance since the guest OS can make some optimizations by knowing it is running within VirtualBox.

Before installing the guest additions, you will need the linux kernel headers and the basic developer tools. On Ubuntu, you can easily install these like so:

$ sudo apt-get install linux-headers-$(uname -r) build-essential dkms

You can upload both public and private boxes to this service.

»Packaging the Box

Packaging the box into a box file is provider-specific. Please refer to the provider-specific documentation for creating a base box. Some provider-specific guides are linked to towards the top of this page.

»Distributing the Box

You can distribute the box file however you would like. However, if you want to support versioning, putting multiple providers at a single URL, pushing updates, analytics, and more, we recommend you add the box to HashiCorp’s Vagrant Cloud.

»Testing the Box

To test the box, pretend you are a new user of Vagrant and give it a shot:

$ vagrant box add --name my-box /path/to/the/new.box...$ vagrant init my-box...$ vagrant up...

If you made a box for some other provider, be sure to specify the --provider option to vagrant up. If the up succeeded, then your box worked!

»Virtual Machine

The virtual machine created in VirtualBox can use any configuration you would like, but Vagrant has some hard requirements:

  • The first network interface (adapter 1) must be a NAT adapter. Vagrant uses this to connect the first time.
  • The MAC address of the first network interface (the NAT adapter) should be noted, since you will need to put it in a Vagrantfile later as the value for config.vm.base_mac. To get this value, use the VirtualBox GUI.

Other than the above, you are free to customize the base virtual machine as you see fit.

»To install via the GUI:

Next, make sure that the guest additions image is available by using the GUI and clicking on “Devices” followed by “Install Guest Additions”. Then mount the CD-ROM to some location. On Ubuntu, this usually looks like this:

$ sudo mount /dev/cdrom /media/cdrom

Finally, run the shell script that matches your system to install the guest additions. For example, for Linux on x86, it is the following:

$ sudo sh /media/cdrom/VBoxLinuxAdditions.run

If the command succeeds, then the guest additions are now installed!

»To install via the command line:

You can find the appropriate guest additions version to match your VirtualBox version by selecting the appropriate version here. The examples below use 4.3.8, which was the latest VirtualBox version at the time of writing.

wget http://download.virtualbox.org/virtualbox/4.3.8/VBoxGuestAdditions_4.3.8.isosudo mkdir /media/VBoxGuestAdditionssudo mount -o loop,ro VBoxGuestAdditions_4.3.8.iso /media/VBoxGuestAdditionssudo sh /media/VBoxGuestAdditions/VBoxLinuxAdditions.runrm VBoxGuestAdditions_4.3.8.isosudo umount /media/VBoxGuestAdditionssudo rmdir /media/VBoxGuestAdditions

If you did not install a Desktop environment when you installed the operating system, as recommended to reduce size, the install of the VirtualBox additions should warn you about the lack of OpenGL or Window System Drivers, but you can safely ignore this.

If the commands succeed, then the guest additions are now installed!

»Packaging the Box

Vagrant includes a simple way to package VirtualBox base boxes. Once you’ve installed all the software you want to install, you can run this command:

$ vagrant package --base my-virtual-machine

Where “my-virtual-machine” is replaced by the name of the virtual machine in VirtualBox to package as a base box.

It will take a few minutes, but after it is complete, a file “package.box” should be in your working directory which is the new base box. At this point, you’ve successfully created a base box!

»Raw Contents

This section documents the actual raw contents of the box file. This is not as useful when creating a base box but can be useful in debugging issues if necessary.

A VirtualBox base box is an archive of the resulting files of exporting a VirtualBox virtual machine. Here is an example of what is contained in such a box:

$ tree.|-- Vagrantfile|-- box-disk1.vmdk|-- box.ovf|-- metadata.json
0 directories, 4 files

In addition to the files from exporting a VirtualBox VM, there is the “metadata.json” file used by Vagrant itself.

Also, there is a “Vagrantfile.” This contains some configuration to properly set the MAC address of the NAT network device, since VirtualBox requires this to be correct in order to function properly. If you are not using vagrant package --base above, you will have to set the config.vm.base_mac setting in this Vagrantfile to the MAC address of the NAT device without colons.

When bringing up a VirtualBox backed machine, Vagrant imports the “box.ovf” file found in the box contents.

»Multi-Machine VAGRANTFILE CONFIGURATIONS
(( took me some trial and error to find the proper syntax/way to specify it to do what I wanted ))

Vagrant is able to define and control multiple guest machines per Vagrantfile. This is known as a “multi-machine” environment.

These machines are generally able to work together or are somehow associated with each other. Here are some use-cases people are using multi-machine environments for today:

  • Accurately modeling a multi-server production topology, such as separating a web and database server.
  • Modeling a distributed system and how they interact with each other.
  • Testing an interface, such as an API to a service component.
  • Disaster-case testing: machines dying, network partitions, slow networks, inconsistent world views, etc.

Historically, running complex environments such as these was done by flattening them onto a single machine. The problem with that is that it is an inaccurate model of the production setup, which can behave far differently.

Using the multi-machine feature of Vagrant, these environments can be modeled in the context of a single Vagrant environment without losing any of the benefits of Vagrant.

Defining Multiple Machines

Multiple machines are defined within the same project Vagrantfile using the config.vm.define method call. This configuration directive is a little funny, because it creates a Vagrant configuration within a configuration. An example shows this best:

Vagrant.configure("2") do |config|  config.vm.provision "shell", inline: "echo Hello"
  config.vm.define "web" do |web|    web.vm.box = "apache"  end
  config.vm.define "db" do |db|    db.vm.box = "mysql"  endend

As you can see, config.vm.define takes a block with another variable. This variable, such as web above, is the exact same as the config variable, except any configuration of the inner variable applies only to the machine being defined. Therefore, any configuration on web will only affect the web machine.

And importantly, you can continue to use the config object as well. The configuration object is loaded and merged before the machine-specific configuration, just like other Vagrantfiles within the Vagrantfile load order.

If you are familiar with programming, this is similar to how languages have different variable scopes.

When using these scopes, order of execution for things such as provisioners becomes important. Vagrant enforces ordering outside-in, in the order listed in the Vagrantfile. For example, with the Vagrantfile below:

Vagrant.configure("2") do |config|  config.vm.provision :shell, inline: "echo A"
  config.vm.define :testing do |test|    test.vm.provision :shell, inline: "echo B"  end
  config.vm.provision :shell, inline: "echo C"end

The provisioners in this case will output “A”, then “C”, then “B”. Notice that “B” is last. That is because the ordering is outside-in, in the order of the file.

If you want to apply a slightly different configuration to multiple machines, see this tip.

Controlling Multiple Machines

The moment more than one machine is defined within a Vagrantfile, the usage of the various vagrant commands changes slightly. The change should be mostly intuitive.

Commands that only make sense to target a single machine, such as vagrant ssh, now require the name of the machine to control. Using the example above, you would say vagrant ssh web or vagrant ssh db.

Other commands, such as vagrant up, operate on every machine by default. So if you ran vagrant up, Vagrant would bring up both the web and DB machine. You could also optionally be specific and say vagrant up web or vagrant up db.

Additionally, you can specify a regular expression for matching only certain machines. This is useful in some cases where you specify many similar machines, for example if you are testing a distributed service you may have a leader machine as well as a follower0follower1follower2, etc. If you want to bring up all the followers but not the leader, you can just do vagrant up /follower[0-9]/. If Vagrant sees a machine name within forward slashes, it assumes you are using a regular expression.

»Communication Between Machines

In order to facilitate communication within machines in a multi-machine setup, the various networking options should be used. In particular, the private network can be used to make a private network between multiple machines and the host.

»Specifying a Primary Machine

You can also specify a primary machine. The primary machine will be the default machine used when a specific machine in a multi-machine environment is not specified.

To specify a default machine, just mark it primary when defining it. Only one primary machine may be specified.

config.vm.define "web", primary: true do |web|  # ...end

»Autostart Machines

By default in a multi-machine environment, vagrant up will start all of the defined machines. The autostart setting allows you to tell Vagrant to not start specific machines. Example:

config.vm.define "web"config.vm.define "db"config.vm.define "db_follower", autostart: false

When running vagrant up with the settings above, Vagrant will automatically start the “web” and “db” machines, but will not start the “db_follower” machine. You can manually force the “db_follower” machine to start by running vagrant up db_follower.

Vagrantfile example I used



Vagrantfile example I used to bring up two virtual machines ( one with Fedora Linux referenced as client and one with Windows Server referenced as windowsserver in the Vagrantfile and in the ansible playbook provision_fedora_vagrantbox.yml) using Virtualbox as the provider.

The Fedora VM is also being provisioned with Ansible using the file “provision_fedora_vagrantbox.yml” (see below)

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.define "client" do |client|
    client.vm.box = "generic/fedora33"
    client.vm.hostname = "fedoraclient"
    client.vm.network :private_network, ip: "10.10.10.10"
    client.vm.provider "virtualbox" do |vb|
                  vb.memory = "2048"
                  vb.cpus = 2 
          end
    client.vm.provision "ansible" do |ansible|
    ansible.playbook = "provision_fedora_vagrantbox.yml"
  end
end

  config.vm.define "windowsserver" do |windowsserver|
    windowsserver.vm.box = "gusztavvargadr/windows-server"
    windowsserver.vm.hostname = "server01"
    windowsserver.vm.network :private_network, ip: "10.10.10.11"
    windowsserver.vm.provider "virtualbox" do |vb|
                      vb.memory = "2048"
                      vb.cpus = 2
          end
  end
end

provision_fedora_vagrantbox.yml

---
  - hosts: client
    become: true
    tasks:
    - name: Disable selinux
      selinux:
        state: disabled

»Explore Other Providers

Vagrants plugin system allows You to plug in other providers not supported OOB for example Microsoft Azure ( see blog article below from Scott Lowe )
https://blog.scottlowe.org/2017/12/11/using-vagrant-with-azure/

Terraform

Introduction to Infrastructure as Code with Terraform | Terraform – HashiCorp Learn

Infrastructure as code (IaC) tools allow you to manage infrastructure with configuration files rather than through a graphical user interface. IaC allows you to build, change, and manage your infrastructure in a safe, consistent, and repeatable way by defining resource configurations that you can version, reuse, and share.

Terraform is HashiCorp’s infrastructure as code tool. It lets you define resources and infrastructure in human-readable, declarative configuration files, and manages your infrastructure’s lifecycle. Using Terraform has several advantages over manually managing your infrastructure:

  • Terraform can manage infrastructure on multiple cloud platforms.
  • The human-readable configuration language helps you write infrastructure code quickly.
  • Terraform’s state allows you to track resource changes throughout your deployments.
  • You can commit your configurations to version control to safely collaborate on infrastructure.

»Manage any infrastructure

Terraform plugins called providers let Terraform interact with cloud platforms and other services via their application programming interfaces (APIs). HashiCorp and the Terraform community have written over 1,000 providers to manage resources on Amazon Web Services (AWS), Azure, Google Cloud Platform (GCP), Kubernetes, Helm, GitHub, Splunk, and DataDog, just to name a few. Find providers for many of the platforms and services you already use in the Terraform Registry. If you don’t find the provider you’re looking for, you can write your own.

»Standardize your deployment workflow

Providers define individual units of infrastructure, for example compute instances or private networks, as resources. You can compose resources from different providers into reusable Terraform configurations called modules, and manage them with a consistent language and workflow.

Terraform’s configuration language is declarative, meaning that it describes the desired end-state for your infrastructure, in contrast to procedural programming languages that require step-by-step instructions to perform tasks. Terraform providers automatically calculate dependencies between resources to create or destroy them in the correct order.

Terraform deployment workflow
https://learn.hashicorp.com/tutorials/terraform/infrastructure-as-code?in=terraform/aws-get-started

To deploy infrastructure with Terraform:

  • Scope – Identify the infrastructure for your project.
  • Author – Write the configuration for your infrastructure.
  • Initialize – Install the plugins Terraform needs to manage the infrastructure.
  • Plan – Preview the changes Terraform will make to match your configuration.
  • Apply – Make the planned changes.

»Track your infrastructure

Terraform keeps track of your real infrastructure in a state file, which acts as a source of truth for your environment. Terraform uses the state file to determine the changes to make to your infrastructure so that it will match your configuration.

»Collaborate

Terraform allows you to collaborate on your infrastructure with its remote state backends. When you use Terraform Cloud (free for up to five users), you can securely share your state with your teammates, provide a stable environment for Terraform to run in, and prevent race conditions when multiple people make configuration changes at once.

You can also connect Terraform Cloud to version control systems (VCSs) like GitHub, GitLab, and others, allowing it to automatically propose infrastructure changes when you commit configuration changes to VCS. This lets you manage changes to your infrastructure through version control, as you would with application code.

Vagrant vs. Terraform

Vagrant and Terraform are both projects from HashiCorp. Vagrant is a tool focused for managing development environments and Terraform is a tool for building infrastructure.

Terraform can describe complex sets of infrastructure that exist locally or remotely. It is focused on building and changing that infrastructure over time. The minimal aspects of virtual machine lifecycle can be reproduced in Terraform, sometimes leading to confusion with Vagrant.

Vagrant provides a number of higher level features that Terraform doesn’t. Synced folders, automatic networking, HTTP tunneling, and more are features provided by Vagrant to ease development environment usage. Because Terraform is focused on infrastructure management and not development environments, these features are out of scope for that project.

The primary usage of Terraform is for managing remote resources in cloud providers such as AWS. Terraform is designed to be able to manage extremely large infrastructures that span multiple cloud providers. Vagrant is designed primarily for local development environments that use only a handful of virtual machines at most.

Vagrant is for development environments. Terraform is for more general infrastructure management.

Getting Started with Terraform:
https://learn.hashicorp.com/terraform

Links

https://www.vagrantup.com/docs

https://www.terraform.io/docs/index.html

https://www.packer.io/docs

Episode 97 – DbVisualizer Pro

https://www.dbvis.com/

DbVisualizer is a feature rich, intuitive multi-database tool for developers, analysts and database administrators, providing a single powerful interface across a wide variety of operating systems. With its easy-to-use and clean interface, DbVisualizer has proven to be one of the most cost effective database tools available, yet to mention that it runs on all major operating systems and supports all major RDBMS that are available. Users only need to learn and master one application. DbVisualizer integrates transparently with the operating system being used.

DbVis Software offers only one product and tries to make it the best possible without offering a vast variety of versions/editions of the same software (many times with reduced set of features as You will see with some of the competitors offers ) and without any additional set of tools to complete the package.

Sometimes less is more and doing one thing and doing it right even if not “absolutely perfect” while being much more affordable to its customers is a Win-Win in my opinion.

Supported Databases

Is your favourite database missing?
DbVisualizer supports most databases with the generic database driver.

Installation and Support Documentation
( Knowledge base, Users Guide)

DbVisualizer is a multi platform tool ( based on Java) able to run on nearly any platform ( Windows , Macintosh , Linux , UNIX ) which makes it platform agnostic in this sense.

Another good thing is that there is only ONE version/installer for both Free and Pro (paid) versions. Pro features are enabled via a license key from within the application itself.

It has an extensive Users Guide ( 430 some pages) kept up to date with each release / new version which is available as an online version which is a browsable HTML and a PDF compiled version for those who prefer a more book / admin manual kind of experience ( I am one of those people)

The above is reinforced with an active Community Forum and an extensive Knowledge Base.

Competitors

While many other companies also offer similar tools to DbVis Software’s DbVisualizer Pro from Navicat to Quest to DbForge more often than not the competitive offers from the mentioned companies might not be the best choice for everyone (price or missing features between editions/software sets)

  • Instead of a single version of the tool/solution which does it all for MySQL, SQL,Oracle, etc. what many of the competitors do instead is creating separate tools for separate segments of the market/customers with the aim to reduce cost/price for the customer while offering a limited feature set of the core application.

    the above is mostly true for DbForge and less to Navicat

    DbForge does not offer to my knowledge a single application which does it all ( work with all databases instead of having to purchase for SQL, MySQL, Oracle, etc.)
    (( I tried to find a single application which is the most similar to Navicat 15 Premium and DbVis Software’s DbVisualizer Pro but I could not figure it out on DbForge’s website myself … ))

    Navicat in the other hand offers its Premium version which in my opinion is the best equivalent of DbVis Software’s DbVisualizer Pro however for a much higher price.
  • The competitors many times offer a lot of additional tools for customers working with Databases both from Navicat and DbForge ( Comparison tools, Dummy Data Creators, Data Compare, Query Builder, Navicat Monitor and Data Modeler) again as a separate product for a separate price instead of baking it all into one unified single application which does all what it can/possible. They do bundle them into packs to offer even more discount on them most of the time.
    I guess it is just another type of business model.

    Sometimes certain vendors like Quest only focuses on one set of Databases (SQL,Oracle) Vs trying to provide tools for all available out there ( from Quest: Toad for Oracle , Toad for SQL Server, SQL Navigator (for Oracle 12c) )

    If You need those extra bits You need to pay for them as they are not in the core utilities provided.

    Many times customers need those extra tools apart from the core utility and they are willing to pay for it.

I leave the links to both Navicat 15 Premium‘s feature set here together with DbVis Software’s DbVisualizer Pro feature set here for you to compare them both and decide which one fits best for your use case / scenario.

dbforge_studio_mysql_Data_View

Image 1 of 2

Key features

Connect to all major databases with a smart and powerful database tool built for efficiency. Improve your productivity and decrease time to learn by using one single tool for all your databases and OSes.

Multi-database and cross-platform Database Management Tool

  • Work with multiple database brands and perform many development and administration tasks from a single tool
  • Easily explore your databases
  • Automatic graphs of primary/foreign-key constraints

Visually create and manage the database

  • Construct and manage database objects with ease
  • Multiple object action executions
  • Create, Compile and Run functions and procedures
  • Object scripting
  • Visually compare table and result set data
  • Powerful data editing

Write and execute SQL scripts

  • SQL IDE with syntax highlighting
  • Auto completion
  • Parameterized SQL execution
  • Chart any result set in nice diagrams
  • Tune your queries with integrated explain plans
  • Query builder using drag and drop, no manual SQL required
  • Visually compare SQL scripts

Data and database transfer

  • Export table data in CSV, SQL, HTML, XML,Excel, JSON, Text
  • Export multiple objects in the database including table data or all objects for a specific type
  • Import table data from CSV files into existing table or create table from import data

General productivity

  • Tasks are executed in background allowing for parallel operations
  • Auto save of the workspace
  • Resume where you left off with editors preserved between sessions
  • Flexible workspace based on a tabbed interface supporting customized layout that is saved between sessions
  • Tag DB objects and SQL script files as favorites for instant loads
  • A dark theme (Darcula)

Product Screenshots

toolproperties

Image 1 of 5

Where to Start with Relational Databases and SQL

You can start your journey into the world of Relational Databases and SQL by signing up to an online course ( free or paid) from the many courses out there from the likes of Udemy to LinkedIn Learning or CBT Nuggets this would be the first piece ( knowledge )

( left some links on the bottom of this page) ( They have a lot on the topics of the SQL language and working with Relational Databases to get you started even on the Cloud with Microsoft Azure and Amazon )

Perhaps picking up some literature from Pearsons Introduction to SQL: Mastering the Relational Database Language, 4th edition works best for those who prefer old school books (ebooks) compared to video courses / online training platforms

For me to see it happen on the screen while explained by an instructor is way better and easier to grasp than reading raw words on a piece of paper but YMMV

The second piece You require to start with SQL and Relational Databases apart from acquiring (knowledge) would be some tools to work with practicing your newfound skills and abilities.

You could start by downloading DbVisualizer either the Free limited functionality version or evaluate the Pro version for 21 days in a non-commercial scenario from here.

You can compare the Free Vs Pro edition on this link here.

You could also set up SQLite on Your computer and connect it with DbVisualizer to start practicing immediately.

There is a great article written by Scott A. Adams explaining just how to do that uat this link

https://scottadams26.medium.com/create-your-own-sqlite-database-with-dbvisualizer-b26863c7b08c

Scott also did a great webinar regarding DbVisualizer which worth to look at (see below)

Taking it one step further – a career in Data Science Boost it with a Bootcamp

There is a lot of Data Science related bootcamps out there if You want to take your journey even further after gaining some knowledge ( can also do a bootcamp from zero no problem ) and experience with the tool DbVisualizer and try to get into the profession of a Data Scientist for a living.


What kind of jobs? What job can I do with these skills / skill sets?
(( If You combine database and database languages skills/knowledge with Mathematics and a high level programming language like Python or R You can work in many different Data Science related fields as You will learn below ))

Key Responsibilities of a Data Analyst

The answer to the question “What does a data analyst do?” will vary depending on the type of organization and the extent to which a business has adopted data-driven decision-making practices. Generally speaking, though, the responsibilities of a data analyst typically include the following:

  • Designing and maintaining data systems and databases; this includes fixing coding errors and other data-related problems.
  • Mining data from primary and secondary sources, then reorganizing said data in a format that can be easily read by either human or machine.
  • Using statistical tools to interpret data sets, paying particular attention to trends and patterns that could be valuable for diagnostic and predictive analytics efforts.
  • Demonstrating the significance of their work in the context of local, national, and global trends that impact both their organization and industry.
  • Preparing reports for executive leadership that effectively communicate trends, patterns, and predictions using relevant data.
  • Collaborating with programmers, engineers, and organizational leaders to identify opportunities for process improvements, recommend system modifications, and develop policies for data governance.
  • Creating appropriate documentation that allows stakeholders to understand the steps of the data analysis process and duplicate or replicate the analysis if necessary.

Most Valuable Skills for Data Analysts

Effective data analysts possess a combination of technical skills and leadership skills.

Technical skills include knowledge of database languages such as SQL, R, or Python; spreadsheet tools such as Microsoft Excel or Google Sheets; and data visualization software such as Tableau or Qlik. Mathematical and statistical skills are also valuable to help gather, measure, organize, and analyze data.

Leadership skills prepare a data analyst to complete decision-making and problem-solving tasks. These abilities allow analysts to think strategically about the information that will help stakeholders make data-driven business decisions and to communicate the value of this information effectively. For example, project managers rely on data analysts to track the most important metrics for their projects, to diagnose problems that may be occurring, and to predict how different courses of action could address a problem.

Data Analysis vs. Data Science vs. Business Analysis

The difference in what a data analyst does as compared to a business analyst or a data scientist comes down to how the three roles use data.

  • The data analyst serves as a gatekeeper for an organization’s data so stakeholders can understand data and use it to make strategic business decisions. It is a technical role that requires an undergraduate degree or master’s degree in analytics, computer modeling, science, or math.
  • The business analyst serves in a strategic role focused on using the information that a data analyst uncovers to identify problems and propose solutions. These analysts typically earn a degree in a major such as business administration, economics, or finance.   
  • The data scientist takes the data visualizations created by data analysts a step further, sifting through the data to identify weaknesses, trends, or opportunities for an organization. This role also requires a background in math or computer science, along with some study or insight into human behavior to help make informed predictions.

At startups and other small organizations, it is not uncommon for a data analyst to take on some of the predictive modeling or decision-making responsibilities that may otherwise be assigned to a data scientist.   

Many courses – nanodegrees (paid) and non-nanodegree (free) courses from udacity

Intro to Data Science (free)
Intro to Data Analyst (free)
Intro to Relational Databases (free)

Data Analyst Nanodegree Program
Programming for Data Science with Python Program
Become a Data Scientist Nanodegree

Disclosure: 

the author’s work on this article was supported by DbVisualizer. The author is responsible for the content produced, including any errors or omissions.

Links

https://scottadams26.medium.com/create-your-own-sqlite-database-with-dbvisualizer-b26863c7b08c

https://www.sqlite.org/index.html

https://www.cbtnuggets.com/it-training

https://www.udemy.com/course/introduction-to-relational-database-and-sql/

https://www.pearson.com/store/p/introduction-to-sql-mastering-the-relational-database-language/P100000789637/9780321305961

https://www.g2.com/products/dbvisualizer/reviews

https://sourceforge.net/s/dbvisualizer/

https://www.capterra.com/p/197030/DbVisualizer/

https://www.trustradius.com/products/dbvisualizer/reviews

https://crozdesk.com/software/dbvisualizer