Network Configuration management (CM) is a systems engineering process created to maintain and establish consistency for products. It's didn't actually originate from the IT industry but it has been widely adopted for use in it. The goal of CM is to systematically handle changes to a system in a way that it maintains integrity over time. For IT people, we generally refer to CM in the context of server configuration management.

Of course, automation plays a central role in the process. Using scripts and specific languages and features of various tools we can achieve our goal.

Back to book index.

Comparison of Network Configuration Management Techniques

We’ve learned that the main idea behind SDN is the separation of various planes within the network. We also now know that the main planes are the control plane and the forwarding/data plane of the networking devices.

Activities such as setting the IP address of devices and basic device configuration, such as remote access configuration, are performed using the management plane of the network. Monitoring activities in the network, such as when interfaces are in a certain state, is also done using the management plane.

Like other processes discussed under SDN, the management plane has also evolved over time. The method of viewing logs from device to device by network managers can now be programmed to send notifications when a certain activity happens.

FIG 27.1 – Data plane vs control plane vs management plane

In Figure 27.1 above, the management plane is where the configuration management occurs.

Traditional Campus Device Management

Traditionally, the management plane of campus devices was in the same chassis like other planes (control and data plane). The management plane is primarily involved with two things:

• Configuration Setup
• Device Monitoring

Configuration Setup

Configuration setup involves basic device configuration such as IP addresses, interface configurations, and even remote access setup (and of course disaster recovery). Items such as access-lists are also considered a part of the management plane, although considering SDN, they can also be carried out in the control plane of the router.

In traditional campus devices, this is done manually across all devices. The assignment of items, such as IP addresses, could be automated using DHCP, however, the rest of the configuration is done manually.

Initial configuration setup is probably one of the most repetitive tasks in network management. Performing access list configuration and setting up SSH in all devices to make sure it is done perfectly can be quite cumbersome.

In order to automate this process, startup configurations could be stored in a remote TFTP server. This means that when a networking device is starting up, we only need to configure the TFTP server and specify that the starting configuration will be obtained from this server. In order to change the starting configurations, it was first modified in the TFTP server and then in each device to ensure that the starting configurations have been updated.

FIG 27.2 – Traditional configuration management from the TFTP server

Yet again, a shell script could be written to help in configuration from a server to all routers in the network. However, Cisco DNA Center presents a more effective solution for this problem.

Device Monitoring

Monitoring of device information and status has traditionally been performed using the Simple Network Management Protocol (SNMP).SNMP is defined in IETF’s RFC 1098 (https://tools.ietf.org/html/rfc1098). This has been quite effective for existing networks for quite some time. It allowed network managers to troubleshoot problems and carry out basic analysis of network activity and ensured logging of activities.

However, SNMP has exhibited significant drawbacks over the years. First, SNMP messages are static and cannot be customized by the network manager. This affects network managers significantly because they need to use a specific type of information in their SNMP messages to enable better management and troubleshooting of problems in the network. It also means if special applications are rolled out in a network, the customization of everything running in the network would be extremely difficult when using SNMP.

SNMP has also been criticized as a security threat as far as networks are concerned. SNMP has an almost complete view of what is happening in the network. This enables it to log information after an event has occurred. It uses SNMP community string to allow access to these details and the management plane of devices. If someone manages to get the correct community string of devices, they would be able to gain access to the requested resource, such as the devices data. SNMPv3 improves this by using a username and a password, instead of the community string, to gain access to the resources.

FIG 27.3 – SNMP security concern

Another downside of SNMP is that it does not separate configuration information and state information. Configuration management plays a key role in the management plane of the network. With SNMP, a network manager is unable to differentiate the logs between configuration information (e.g. when getting the configuration of a specific interface like an IP address) and state information (e.g. when a specific interface goes down which disables some routing functionality). This information will be considered same in SNMP logs. Once again, Cisco DNA Center provides a comprehensive solution to this problem.

Configuration Management Tools

Configuration Management is a system engineering process designed to simplify the management of an infrastructure as its scale increases. This enables automation in configuration and maintenance of a company’s technological assets.

Configuration management involves the automation of commands, automation in deploying applications in an infrastructure, and monitoring of these applications and services in the infrastructure. Configuration management helps in simplifying many processes that have become time consuming in the past, like making sure the packages are up to date accordingly with the applications running on servers.

Configuration management tools offer many benefits including:

• Automation provides configuration provisioning and deployment of application and infrastructure.
• Programming knowledge is not required—use of declarative model (intent): not scripting!
• Leverage software development practices for deployments, including version control and testing.

There are three common tools are used for the configuration management field, and which are mentioned specifically in the exam syllabus. These are:

• Ansible
• Puppet
• Chef

FIG 27.4 – Configuration management tools server in operation

From the networking perspective, we can use above mentioned configuration tools to define and enforce configuration related to system level operations (for example authentication logging, reimaging), interface level configuration (for example VLAN, QoS, security), routing configuration (for example EIGRP, ISIS, BGP specifications), and so on.

These tools are often referred to as DevOps tools, and they are more specifically used for configuration management and automation tools that happen to be used by those organizations that have implemented some form of DevOps practices, like version control and testing.

Our main focus will be on Ansible because it is used by Cisco as its main configuration management tool. Its capabilities and drawbacks will be explored along with discussing some basic concepts of configuration management tools. Some examples based on the Cisco DevNet Sandbox will also be discussed.

After this, an overview of the capabilities of Puppet and Chef will be presented, two other industry leaders when it comes to configuration management.

Ansible

Ansible prides itself in presenting the simplest way to automate IT Infrastructure. It is an open source configuration management tool, first introduced in 2012 by Ansible Works and was acquired by Red Hat foundation in 2015. Ansible has penetrated quickly into the market due to the following two major factors.

• Ansible is the simplest of the three configuration management tools mentioned previously. Understanding of YAML and Python is enough to get started with Ansible as it does not require some prior knowledge.
• Ansible uses Python which is built-into most Unix systems. Ansible has found its application in the DevOps world quite easily because most systems in DevOps use Linux.

Ansible Concepts

As Ansible is a configuration management tool, it can run commands on remote hosts and perform specific tasks based on a concept called plays (which will be discussed later). Ansible uses SSH functionality within the remote hosts (servers or networking devices) and can carry out commands and plays in them.

There are two methods a user can use to access a remote host via SSH.

The first is by using a password. In this method, the remote host configures a password for SSH access (port 22 by default). Any client that wants to access the remote host enters the password in order to log into the system via SSH. This approach has one significant problem. Anybody that has the SSH password will have complete access to the host. A hacker only needs to get the password, and he can access the system without any restriction.

The second method is through SSH keys. This method is used by Ansible and is considered to be a more secure way to login via SSH. This method works because it is easier to limit the access to the system as compared to using passwords, where anyone with a password can access the system. SSH keys are unique and identify a host.

Following are the steps to login via SSH keys:

1. Keys are generated in the SSH client machine.
2. The key is added to the host machine that is accessed via SSH. It is added to a file called known_hosts. This lists the keys to the machine it is familiar with.
3. When a login attempt is made from the machine containing the keys, there will not be any password prompt.

A major advantage of this method is that only those machines are able to access the file via SSH that have their keys enlisted in the known_hosts file.

In our case, Ansible will be the client trying to access the remote hosts, which include servers, networking devices, or any other host accessible via SSH. Below is a pictorial representation of how Ansible accesses the remote hosts.

Being natively agentless significantly lowers the barrier to entry from an automation perspective. Because Ansible is agentless, it can integrate and automate any device using any API. For example, integrations can use REST APIs, NETCONF, SSH, or even SNMP, if desired.

FIG 27.5 – After the server with Ansible has access to a host, commands can be executed on it

The key Ansible components are:

• Modules
• Playbooks
• Inventory
• Configuration File (ansible.cfg)

Modules

Modules are essentially plugins for Ansible that simplify the way tasks are performed. These are python scripts and reduce complexity.

Whenever a playbook and a module are running, the module is executed on the remote host. For example, if we want to ping all hosts, we can use the ping module and decide what to do with the output.

There are also networking modules that have been developed to simplify networking processes. A lot of Cisco specific modules have been made for Ansible and can be accessed at: https://www.ansible.com/integrations/networks/cisco

These modules allow a lot of automation in a network. Previously, Cisco based networking controllers were discussed. Most of them have Ansible modules that allow automation of specific activities across the entire SDN domain.

Plays

A play is a specific task performed by Ansible. For example, when a new application is deployed in a host and a dependency update is needed, a play can be made to check the dependencies required by the application. If the required dependency is already running and updated, it does not make any changes in the application. It updates the dependency only when needed.

Therefore, play is a list of instructions developed to perform a specific task on host machines. These instructions are written in a Domain Specific Language (DSL)

Playbook

A collection of plays (tasks) is called a Playbook. There can be more than one related task that should be accomplished, each task is a Play and stored inside the Playbook.

For instance, in the above example of Plays introduction, package updating before deploying the application is considered a play. If a web service needs to be restarted before deploying the updated packages, then the process of restarting the web service can be considered a second play. These two plays together constitute a playbook.

Ansible are written in YAML. YAML files are written in .yml or .yaml extension.

Inventory

An inventory is a list containing groups of hosts. These are very useful in large infrastructures.

The hosts in an inventory are identified by their addresses, either IP addresses or hostnames. The address used to identify hosts must be reachable from the Ansible server via SSH.

Inventories not only store lists of hosts but they also allow their appropriate collection known as a group. This can be very useful when the servers have specific functionality. For instance, if some servers are used to host the database, then other entirely different servers are used to host the application.

Therefore, an inventory can have two groups; one with database servers as its host and the other with application servers as its host. One thing should be kept in mind that it is not mandatory for a host to be in only one group. If there is an application and a database server, then it can appear in both groups.

These groups not only classify servers but also the networking elements. For instance, all networking devices can be classified as groups based on their subnets, their locations, or type of devices. For example, there can be a group for all the Nexus devices when running Ansible.

Configuration Files

Ansible configuration is kept in .cfg files. These files contain details such as the default inventory file, roles etc.

By default, changes can be made using the configuration file called ansible.cfg. There are various reasons DevOps engineers want to change this file. For instance, changing the directory settings of how Ansible runs, how Ansible library runs, the modules to use, and default timeout. This is very useful while carrying out custom configuration or deploying solutions.

Ansible Lab

The following structure for Ansible will be established:

FIG 27.6 – Ansible LAB structure

The structure shown in the above figure will be used for this example. There will be two hosts namely host1 and host2.

Goal

The goal for this lab is to remotely control host1 and host2 from the server by using Ansible basic commands. A directory and a sample file example.txt, with “Hello world” text, will be created in each host.

To install Ansible, follow the Ansible installation guide available under the support documents at ansible.com.

Setup SSH

By default, Ansible communicates with its hosts using SSH by using private keys. Let us see how this is done.

The keys are generated on the Ansible server using the following command:

ssh-keygen

This would prompt the user to enter the key in the file, which should be the .ssh/id_rsa file located in users’ home directory.

The content of id_rsa file is copied using the following command:

cd ~/.ssh

cat id_rsa.pub

This command gives the public key that has been generated on the Ansible server.

The user will now login to the remote host and open the ~/.ssh/known_hosts file. The keys copied in the previous step will be pasted in this file.

This will setup an SSH process. The SSH connection can be tested by the following command:

ssh user@address

This process will provide access to the remote hosts’ terminal and can be repeated for all the hosts that will be configured using Ansible.

Setup the Inventory

Now is the time to configure any hosts that we want to. In our case, vagrant is used, and both hosts are virtual machines. The IP address of the first host is 192.168.43.7 and the IP address of the second host is 192.168.43.158

The setup looks as follows:

FIG 27.7 – Ansible LAB setup

Let us create a directory called ansible-lab and make a new file called hosts inside this directory. The inventory will be stored in this file. The hosts file will look like:

[ansible-lab-hosts]

192.168.43.7

192.168.43.158

This creates a group called ansible-lab hosts with addresses 192.168.43.7 and 192.168.43.158 as hosts in the group.

Write the playbook

Following commands will add “Hello world” to example.txt file located in the ansible directory of hosts.

mkdir ansible

cd ansible

touch example.txt

echo Hello world > example.txt

A file named hello-world.yml will be created in the same directory where the host file is located by copying & pasting the following:

—

– hosts: ansible-lab-hosts

remote_user: vagrant

tasks:

– name: create the ansible directory

file:

path: ~/ansible

state: directory

 

– name: create the file example.txt in the ansible directory

file:

path: ~/ansible/example.txt

state: touch

 

– name: insert “Hello world to the example.txt file”

copy: content=”Hello world” dest=~/ansible/example.txt

Review of playbook

The first line, — indicates that a playbook has been started.

The playbook discussed here consists of only one play. Ansible is quite indentation sensitive, so the first level always denotes the plays. In our case, because we have only one element at the first indentation level, there is only one play.

The following two lines:

– hosts: ansible-lab-hosts

remote_user: vagrant

The above commands set up the remote connection. It shows the group that ansible-lab-hosts will be used which consists of two hosts, 192.168.43.7 and 192.168.43.158. The user in both hosts is called vagrant which is specified in the second line.

Now, let us look at the tasks. Each play consists of one or more tasks. A play has a goal, and this goal can be achieved by running several tasks. In our case, there are three tasks in one play.

1. The first task is to create the ansible directory which creates the directory called ‘ansible’. This task uses the file module which helps in specifying the path which we want to be affected and the required state. If the state is a directory, the specified path will create a directory. In our case, the directory created will be called ‘ansible’.
2. The second task is to create the example.txt file. This task again uses the file from in the previous task. The difference is in the state. In the previous task, a directory was created so the state was a directory. In this task, a .txt file is created so the state will be touch (read up Linux for more information on this if you wish).
3. The final task prints “Hello world” in the example.txt file. It uses the copy module which takes two parameters—content and dest (the destination file). In our case, the content is “Hello world”, and the destination file is ansible.txt.

To execute this playbook, following commands will run:

ansible-playbook -i hosts hello-world.yml

This will show the following output:

[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

/usr/local/lib/python2.7/dist-packages/requests-2.22.0-py2.7.egg/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.

warnings.warn(warning, RequestsDependencyWarning)

 

PLAY [ansible-lab-hosts] ***********************************************************************************************************************************************************************************

 

TASK [Gathering Facts] *************************************************************************************************************************************************************************************

ok: [192.168.43.158]

ok: [192.168.43.7]

 

TASK [create the ansible directory] ***********************************************************************************************************************************************************************

changed: [192.168.43.7]

changed: [192.168.43.158]

 

TASK [create the file example.txt in the ansible directory] ************************************************************************************************************************************************

changed: [192.168.43.7]

changed: [192.168.43.158]

 

TASK [insert “Hello world to the example.txt file”] ********************************************************************************************************************************************************

changed: [192.168.43.158]

changed: [192.168.43.7]

 

PLAY RECAP *************************************************************************************************************************************************************************************************

192.168.43.158        : ok=4     changed=3  unreachable=0     failed=0   skipped=0  rescued=0  ignored=0

192.168.43.7          : ok=4     changed=3  unreachable=0     failed=0   skipped=0  rescued=0  ignored=0

The addresses will change according to remote hosts.

This should just give you an overview of how Ansible works.

Puppet

Puppet is an open source tool for configuration management developed in Ruby. It has been employed since 2005, long before Ansible was built.

Puppet takes a bit longer to install than Ansible. For Ansible, an SSH connection to the remote host was required.

FIG 27.8 – Puppet Architecture

Puppet operates in a client-server architecture. In order to run it, Puppet Server should be installed on the server(s) that requires configuration management control. Also, the Puppet Agent should be installed in each host managed device (node) and a Puppet Master (server). The Puppet Master typically runs on a separate dedicated server. When there are new configurations set on the master node, the rest of the Agents pull these configurations from the master. In Ansible, the master pushed the configurations into the hosts. However, in Puppet, the hosts are more involved in the deployment of configurations.

To carry out configurations in Puppet, Ruby’s Domain Specific Language (DSL) should be understood. Ruby’s DSL may take a bit longer to understand as compared to YAML. However, YAML is more sensitive to indentation as compared to DSL.

The user interface that Puppet offers is, however, very clean. It allows better management of Agents. An Enterprise Ansible version may provide interface simplicity but still Puppet is considered simpler and more effective.

Chef

Chef is also a configuration management tool built in Ruby. It emerged in 2009 as Puppet’s competitor and, like Puppet, uses Ruby’s DSL for carrying out the configurations in files.

Chef contains features of both Ansible and Puppet in terms of how its deployment is carried out. It has a client-server architecture, like Puppet, but still uses SSH to deploy the configurations in the hosts like Ansible. The nodes in Chef must be configured periodically to check the master for new configurations that need to be installed. Unlike Ansible, it is not possible in Chef to make changes from the master to nodes; the nodes always pull the changes from the master.

Chef offers complete customization of configurations. Chef allows the use of both: Ruby Domain Specific Language and embedded Ruby (ERB) templates. These give managers more power to build custom configuration management solutions for their organizations.

However, Chef has been criticized for its installation mechanism. It is recommended to use a bash script for Chef installation which is considerably an insecure way to install packages.

Chef’s architecture is illustrated below:

FIG 27-9 – Chef Architecture

Interpretation of JSON Encoded Data

JSON is abbreviated for JavaScript Object Notation. JSON is a lightweight data format that is used in web services for transmitting data. It is widely used in scripting-based platforms because of its simple format. It is a data exchange format that allows various computers (servers and clients) exchange data among themselves. JSON has gained popularity due to the following factors:

• Simplicity – JSON is quite simple to understand and read. Knowledge of JavaScript is not required to start working with JSON.
• Lightweight – When it comes to networking resources, JSON is very lightweight and resource effective. This gives a huge advantage especially when its data exchange format is used in web-based REST API. The speed technologies used by JSON for data exchange are quite fast.
• Widely supported – Almost all popular programming languages can parse JSON data. This makes it a lot easier for developers as it becomes independent of the platform on which the technology is built.

Understanding interpreting JSON is a rudimental skill in modern day networks and tested in the CCNA exam. A brief introduction to JSON basics is given below.

JSON Formatting

While discussing REST APIs, an analogy of drive-through was presented with their windows being the points of access for specific resources. These windows need a standard means of communication. If the order is placed in English and the person on the drive-through counter speaks Spanish, the customer will have a hard time getting through the order. Similar situation occurs in APIs, communication cannot be established between systems if the format used to expose the system data is different from the format used by the listening system to fetch the data. The format frequently used in REST APIs is JSON, which will be discussed shortly. In other words, the format in which a system exposes its details should be similar to the format in which another system is expecting the details.

The output below illustrates how JSON data appears:

{

“lineCardCount”:”9″,

“series”:”Cisco ASR 1000 Series Aggregation Services Routers”,

“collectionInterval”:”Global Default”,

“collectionStatus”:”Managed”,

“roleSource”:”AUTO”,

“softwareType”:”IOS-XE”,

“lastUpdated”:”2020-03-11 13:35:46″,

“managementIpAddress”:”10.10.22.253″,

“platformId”:”ASR1001-X”,

“memorySize”:”3819298032″,

“errorDescription”:null,

“location”:null,

“bootDateTime”:”2019-11-08 00:14:32″,

“softwareVersion”:”16.3.2″,

“interfaceCount”:”12″,

“tunnelUdpPort”:null,

“id”:”1cfd383a-7265-47fb-96b3-f069191a0ed5″,

“apManagerInterfaceIp”:””,

“associatedWlcIp”:””,

“role”:”BORDER ROUTER”,

“inventoryStatusDetail”:”<status><general code=\”SUCCESS\”/></status>”,

“snmpContact”:””,

“errorCode”:null,

“family”:”Routers”,

“lineCardId”:”d9ee06c9-1d0c-4032-919a-9896b0119567, 999f98b1-19bc-4146-a70c-a465ba449831, 1d21ac54-2aee-4e64-8076-fa7638f11a2e, cbcacc67-5020-4e80-b742-870d81f1b764, b84abe3a-6b3a-4f8f-a6b8-fcce654fe753, 4ebb74b8-2850-4ebd-b288-60b4d1500eb3, c46992ff-fed6-4c3c-8765-a253130090f4, 159a794d-fa02-442e-ac22-25496395934b, 0d54d112-96b3-47b3-8d35-03aa8abd7e0c”,

“instanceUuid”:”1cfd383a-7265-47fb-96b3-f069191a0ed5″,

“type”:”Cisco ASR 1001-X Router”,

“reachabilityStatus”:”Reachable”,

“instanceTenantId”:”5dc444d31485c5004c0fb20b”,

“hostname”:”asr1001-x.abc.inc”,

“tagCount”:”0″,

“upTime”:”35 days, 2:16:57.11″,

“reachabilityFailureReason”:””,

“waasDeviceMode”:null,

“MACAddress”:”00:c8:8b:80:bb:00″,

“serialNumber”:”FXS1932Q1SE”,

“locationName”:null,

“snmpLocation”:””,

“lastUpdateTime”:1583933746136

}

Data Types in JSON

JSON uses 6 data types. First four types (string, number, boolean, and null) are referred to as simple data types. The last two data types (object and array) are referred to as complex data types.

The following data types to represent information:

• Strings
• Numbers
• Boolean
• Null
• Javascript Objects
• Javascript Arrays

String

Any value enclosed in double quotes is a string. For example, the word GigabitEthernet5 in the following code is a string.

{

“name” : “GigabitEthernet5”

}

Numbers

Numbers are decimal digits ranging from 0 to 9. They can be either integers (numbers without decimal points) or floats (numbers with decimal point values).

In the example used in previous sections explaining JSON, words were used to describe numbers, such as “billions”, “millions”, and “thousands”. All these words can be represented as numbers to make JSON data cleaner and easier to comprehend if this data is used for mathematical computations. For example:

{

“CRC Errors” : 105

}

Boolean

A Boolean is a true or false value. Anything that has only two states, on or off, up or down, true or false is considered a Boolean.

In JSON, any key value representing something that is true is written as true and anything that is not true is written as false.  For example:

{

“enabled” : true

}

Null Values

Null values are kept when an item does not have any information. Normally, N/A or Not Applicable is used while filling out forms. Null is equivalent to N/A in JSON and many other programming languages.

In JSON, to represent a Null value, null keyword is used. For instance, if we are looking for a class teacher's name and the class does not have a teacher, we can represent it as null:

{

“msec”: null

}

JavaScript Objects

A JavaScript Object is a collection of key value pairs, which behaves almost like a dictionary. In a dictionary, each word has a definition, meaning, or explanation of the word. For JavaScript Objects, the key is like a word in the dictionary and is used to represent something. The value of the key is like definition of a word.

Note: Every key in a JavaScript Object must be unique, just like words in a dictionary cannot be repeated. This is because each item in an Object is identified by its key, so the ID must be unique.

Similarly, a JavaScript Object will always have a value associated to its key. A JavaScript Object Starts with ( { ) and ends with ( } ). Between these curly brackets, everything needs to be a key value pair, as explained previously.

Let us look at an example of a JavaScript Object:

{

“planet”: “earth”

}

This is a simple JavaScript Object with one element. This element contains the key “planet” and the value “earth”.

If addition of elements is required, a comma (,) is used to add another element as specified in example below:

{

“planet”: “earth”,

“Population”: “seven billion”

}

Objects can have unlimited key value pairings.

Nested Objects, that is, an object inside an object, are also supported in JSON. This is normally done in a case when a key value requires further explanation and another object is needed to define it better.

{

“planet”: “earth”,

“population”: {

“Europe”: “seven hundred and forty one Million”,

“North America”: “five hundred and seventy nine million”,

“Africa”: “one billion, two hundred and sixteen million”,

“Asia”: “four billion, four hundred and sixty three million”,

“Antarctica”: “not available”,

“South America”: “Four Hundred and seventy two million, five     hundred thousand”,

“Oceania”: “forty two million, five hundred thousand”,

“Australia”: “Two million, six hundred thousand”

}

}

The Object above has given population a new meaning by individually defining each continents’ population.

JavaScript Arrays

A JavaScript Array is a list of items which represents a collection of items but is not a key value pair like the JavaScript Object.

[

“Paul”,

“Miriam”,

“Brenda”,

“Gideon”

]

JavaScript Array items are represented by their location (first, second, third…). Therefore, the items in an Array need not to be unique.

A combination of both arrays and objects can exist in JSON data. An example of the planet is mentioned below:

{“planet”: “earth”,

“continents”:

[

{

“name”: “Europe”,

“population”: “seven hundred and forty one Million”

},

{

“name”: “North America”,

“population”: “five hundred and seventy nine million”

},

{

“name”: “Africa”,

“population”: “one billion, two hundred and sixteen million”

},

{

“name”: “Asia”,

“population”: “four billion, four hundred and sixty three million”

},

{

“name”: “Antarctica”,

“population”: “not available”

},

{

“name”: “South America”,

“population”: “Four Hundred and seventy two million, five hundred thousand”

},

{

“name”: “Oceania”,

“population”: “forty two million, five hundred thousand”

},

{

“name”: “Australia”,

“population”: “Two million, six hundred thousand”

}

]

}

In this example, it can be seen how objects and arrays can work together. This is a representation of how JavaScript data looks like. Both JSON Objects and Arrays are utilized to display data properly.

Conclusion

After looking at all these data types, we can simplify our earlier example, and enter the data types where it is appropriate.

{“planet”: “earth”,

“continents”:

[

{

“name”: “Europe”,

“population”: 741000000

},

{

“name”: “North America”,

“population”: 579000000

},

{

“name”: “Africa”,

“population”: 1216000000

},

{

“name”: “Asia”,

“population”: 4463000000

},

{

“name”: “Antarctica”,

“population”: null

},

{

“name”: “South America”,

“population”: 472500000

},

{

“name”: “Oceania”,

“population”: 42500000

},

{

“name”: “Australia”,

“population”: 2600000

}

]

}

This is a good example of how JSON data is written down. Let us try and interpret it:

The information discussed above has the name earth. It has items called continents, and each of the continents has a name and the population. We can derive the name and population of each continent from the JSON data shown above.

For practice, try and interpret the following JSON information:

{

“name”: “Barack Obama”,

“country”: “USA”,

“gender”: “male”,

“married”: true,

“age”: 58,

“wife”: “Michelle”,

“children”: [“Malia”, “Sasha”],

}

JSON Formatting Pitfalls

When sending JSON data between systems, sometimes errors may occur. This is mainly due to formatting of JSON data, especially when JSON information is entered manually.

The following pitfalls will be based on guides set by RFC 8259, “The JavaScript Object Notation (JSON) Data Interchange Format” RFC Standardization Track. The most common pitfalls are listed below:

1. Not closing arrays and objects

According to part two of RFC 8259, JSON-grammar, there is a begin-array symbol ( [ ) and begin-object symbol( { ). If one of the two is used, there must be a corresponding end-array ( ] ) or end-object( } ) symbol.

To avoid this mistake, always count the number of begin-array symbols or begin-object symbols ( [ ), and make sure that it is equal to the number of end-array or end-object symbols ( ] ).

2. Not matching the opening and closing brackets for objects and arrays

Another issue related to JSON arrays and objects is the sequence in which we open and close them. If there is an array and an object inside the array, the symbols will be ( [{ ), an array within an object. Similar sequence should be used while closing them ( }] ). It should be noted that the sequence is reversed while closing them.

An error will occur if an array has an object nested inside it ( [{ ), and is closed like this ( ]} ). The number of beginning and closing symbols should always be the same for arrays and objects.

3. Using single quotes for strings

This is quite easy to figure out. Strings, as mentioned before, are enclosed in double quotes ( “ ). For example, “name” is a valid string in JSON. It would be invalid if it was surrounded by single quotes.

To have a better understanding, please refer to RFC 8259 section 7, “Strings”.

4. Failing to close strings

Strings are the values in quotes, such as “Paul”. A string becomes invalid when it is opened but not closed, such as “Paul. Every opening quote must be accompanied by a closing quote after the string.

5. Using a name separator inside an array element

As mentioned in the JavaScript Arrays section, an array is a list of elements. Each element can be a new object or a literal (false, null, true, object, array, number, string) but each element is a value. Values cannot be separated by a name separator (:), only object elements can be separated by a separator.

Following object is invalid:

[“Paul”, “age”: 23, “country”: “USA”]

This will give an error because the second and third element in the array are values separated by a name separator ( : ). If an array element value is needed as a key: value pair and use the name separator ( : ), you can separate the specific elements into objects, like this:

[“Paul”, {“age”: 23}, {“country”: “USA”}]

6. Not using a name separator inside an object element

An element inside an object cannot exist without a name separator ( : ) because an object is mainly meant for key value pairing, which is the function of a name separator.

We, therefore, cannot have the following:

{ “Paul”, “age”: 23, “country”: “USA” }

The first element “Paul” separated by a name separator is not a key value pairing. The above data can be rewritten as:

{ “name”: “Paul”, “age”: 23, “country”: “USA” }

7. Using a semicolon instead of a colon

This issue is quite common and a lot easier to identify and rectify. The name separator ( : ) which is basically a colon, is used to separate key value pairings inside an object. It can be easily confused with the semicolon ( ; ), especially for someone who has a programming background in JavaScript where each statement ends with a semicolon.

Please note that semicolon is only used inside a string in JSON. Following is an example of this mistake:

{“name”: “Paul”, “country”; “USA”}

In the second element of the object, “country”; “USA” is used. This will cause an error when transmitting the data because the object element has not been separated with a colon as:

{“name”: “Paul”, “country”: “USA”}

Sample Exercise

Let us look at the following JSON samples and figure out which one is valid and why. The answers will be at the end of the questions:

1. Children in a class – We have two children, Paul, 20 years old, and Gideon, 34 years old. We are trying to send their data via JSON to a remote system. Which of the following will be a valid way to send this data.
   1. a.

      [{“name”: “Paul”, “age”: 20}, “name”: “Gideon”, “age”: 34]

   2. b.

      [{“name”: “Paul”, “age”: 20, {“name”: “Gideon”, “age”: 34}]

   3. c.

      [{“name”: “Paul”, “age”: 20, {“name”: “Gideon”, “age”: 34}]

   4. d.

      [{“name”: “Paul”, “age”: 20}: {“name”; “Gideon”, “age”: 34}]

2. User Information – A user has logged into a system and requires his information. Which of the following would be a permissible way to send this information across?
   1. a.

      { “email”: “paul@gmail.com”, “username”: “paul”, “id”: “e33f122a” }

   2. b.

      { “email”: “paul@gmail.com”, “username”: “paul”, “id”: “e33f122a }

   3. c.

      { “email”: “paul@gmail.com”, “paul”, “id”: “e33f122a” }

Answers

1. Correct answer: c
   1. a.    This is wrong because we are using a name-separator ( : ) in an array. Look at the second          element, “name”: “Gideon”, and the third element, “age”: 34. Both are using a name                  separator, and they are not an object element, which is not allowed.
   2. b.    This is wrong because we have an object that has not been closed. As mentioned in pitfall          1, we should always count the number of begin-objects( { ) and make sure they are the              same as the number of end-objects( } ). The number of begin-array( [ ) should also be                the same as end-arrays( ] ). In this case, there are two begin-objects( } ) but only one                closing object( } )
   3. c.     This is correct.
   4. d.    Here, we have used an illegal character, semicolon( ; ). As mentioned in the final pitfall,            the semicolon can easily be confused with a colon ( : ) used to separate key-value                        elements in the object. There is a “name”; “Gideon” element in an object which uses a                semicolon.
2. Correct answer: a
   1. a.    This is correct JSON format. It will be transmitted successfully.
   2. b.    In this example, we have a string that has the opening quotes but closing quotes are                   missing. The final element “e33f122a, is a string without closing quotes. It should be                 written as “e33f122a”.
   3. c.    Here, we have an object element that is not in the correct format. A JSON object is a key           value pair. In the second element for the object, we have only a value, “paul” and not a             key value pair separated by a colon.

For further reference, you can have a look at RFC 8259 standardization online (https://tools.ietf.org/rfc/rfc8259.txt). This is a rather brief RFC standardization document compared to most others and easy to peruse through. If you have time, you can go through it and understand some of the things the standard requires when sending the data.