One of the more challenging issues with wireless sensor nodes is performing the initial configuration required to get them to connect to your wireless network in the first place (in this case I am talking about 802.11 a/b/g/n based WiFi networks). To connect to a WiFi network a sensor needs to have at least two pieces of information - the SSID (network name) and a password to grant access to that network - the problem is how to provide that information to a brand new, just programmed sensor?

The quickest and easiest solution is to include it in the source code for the firmware and have it always connect to the right network and there is a lot of sample code available that does just this. While this is a workable solution for personal projects (just make sure you don't accidentally commit code with your plain text network password in it) it's not very portable - what happens if your WiFi settings change or you want to move the device to another network? The source will need to be modified, re-compiled and the sensor updated with the new flash image.

What I wanted was a solution that:

  • Could be used with no custom tools - just a phone and a web browser should be enough to configure the device.
  • Was portable - it doesn't rely on any ESP8266 features and could be used on other WiFi modules as well.
  • Can be extended to support other configuration options (MQTT server and topic to publish on for example).
  • Provides an interface that is easy to use from code - a Python script could discover all available sensors on the network and reconfigure them programatically.

What I came up with is working fairly well and seems to meet all of the requirements above. It needs a bit more field testing before I release the source code on GitHub but it's worth while describing the implementation here if you are trying to build the same sort of thing.

If you just want an 'out of the box' solution I recommend looking at the WiFiManager library for the ESP8266 Arduino core. It is a complete solution that can just be dropped in to your sketch to provide almost all of the functionality I listed above. The reason I didn't use it myself is because I was interested in how all the underlying bits would work together.

How It Works

My implementation has two modes of operation - configuration and running. Configuration mode is used when the device does not have a valid configuration (the data is blank or it cannot connect to the specified WiFi network) and running mode is used while your sketch is running to allow 'live' updates to the configuration. The implementation uses mDNS to announce the configuration service on the network so devices can easily be discovered.

In both modes a HTTP server is listening on port 80 and configuration is managed through the '/config' URI. Posting configuration values as JSON to '/config' will update the current settings, performing a GET request on the URI will retrieve the current settings in JSON format. When in 'configuration' mode the device runs as an access point (with a recognisable SSID) and runs it's own DNS server to point any domain name to it's IP address, acting as a captive portal. To perform the configuration you simply connect your phone to the devices access point and navigating to any HTTP address will bring up the configuration interface:

IoThing UI

Once you click save the device will store the new WiFi credentials and restart. Because it advertises it's services using mDNS you can then use service discovery to find and communicate with the device and set any additional configuration parameters that are required. In my case I use a Python script to monitor for new sensors which then sets the MQTT server parameter - the sensors will then publish their data to that server where it can be accessed from a single point.

Implementation

Luckily almost all the functionality required to implement this is already provided in the ESP8266 Arduino core libraries. These are not well documented unfortunately, you have to dig into the source code to discover what is available. The libraries I'm using include the ESP8266WiFi library to set up the access point and control connection to the WiFi network, the ESP8266WebServer library to provide the web interface and configuration endpoints and the DNSServer library to implement the captive portal. In this section I will describe how they are all used. First though it's worth looking at a flow chart of how it all fits together.

Configuration Flowchart

When the device first powers up we try and read the current configuration information from the EEPROM and validate it with a checksum - if it passes we assume it is correct and try to connect to the SSID provided. This is done using the standard WiFi library and we try up to three times if the connection attempt fails. If the configuration is not valid or we cannot connect to the network specified in the configuration we enter configuration mode.

In configuration mode the device needs to act as an access point so a phone or tablet can connect to it to access the configuration page. The access point name needs to be easily recognisable so the user can select the appropriate one but still have some uniqueness to avoid clashing with existing network names. This project is part of a larger group of projects I've tentatively called 'IoThing' so I use that as the prefix for the network name - to help avoid clashes I then add a 2 digit number to the name. The ESP8266 conveniently provides a unique chip ID so I just use the last two digits of that number in the name.
This will not guarantee a unique name however (you may be unlucky enough to have two ESPs with similar chip IDs) so as a safety check I scan the available networks and check for clashes, if one is found it waits until that network goes away before actually activating the access point. The code for this is shown below.

//--- SoftAP configuration
IPAddress apIP(192, 168, 4, 1);  
IPAddress netMsk(255, 255, 255, 0);

void setupAccessPoint() {  
  static char szSSID[12];
  sprintf(szSSID, "IoThing %02d", ESP.getChipId() % 100);
  // Set WiFi to station mode and disconnect from an AP if it was previously connected
  WiFi.mode(WIFI_STA);
  WiFi.disconnect();
  delay(100);
  // Make sure the requested SSID is not being used
  bool exists = true;
  while(exists) {
    int n = WiFi.scanNetworks();
    exists = false;
    for (int i=0; i<n; i++) {
      String ssid = WiFi.SSID(i);
      Serial.print("Found SSID ");
      Serial.println(ssid);
      if(strcmp(szSSID, ssid.c_str())==0)
        exists = true;
      }
    if(exists) {
      DMSG("AP '%s' is already in use, waiting.", szSSID);
      delay(5000); // Wait before scanning again
      }
    }
  // Set up the open AP with the given SSID
  WiFi.mode(WIFI_AP);
  WiFi.softAPConfig(apIP, apIP, netMsk);
  WiFi.softAP(szSSID);
  delay(100);
  }

To implement a 'captive portal' (where every domain name is resolved to the IP address of the sensor) we need to configure the DNS server appropriately. Luckily the DNSServer library has full support for this - the code snippet below shows how it is set up. In this case we tell the server never to report an error and that it should respond to all requests (the '*' in the 'start()' function parameters).

DNSServer dnsServer;

void setupDNS() {  
  /* Setup the DNS server redirecting all the domains to the apIP */
  dnsServer.setErrorReplyCode(DNSReplyCode::NoError);
  dnsServer.start(53, "*", apIP);
  }

The next step is to provide the configuration page. Remember that there are two parts to the configuration - the '/config' URL which allows you to read and write the configuration values using JSON and the configuration UI page. In this case I created a single HTML page with embedded javascript - when the page loads it requests the current configuration from '/config' using a JSON request and populates the form fields appropriately. When you save the values it gets the values from the form fields, creates a JSON string and makes a POST request to '/config' to save them.

function onConfig(values) {  
  document.getElementById("ssid").value = values["ssid"] || "";
  showSection("config");
  }

function getJsonResponse(status, payload) {  
  if (status != 200) {
    showSection("error");
    return null;
    }
  try {
    return JSON.parse(payload);
    }
  catch(err) {
    showSection("error");
    }
  return null;
  }

function readConfig() {  
  var url = "http://" + location.hostname + (location.port ? ':' + location.port : '') + "/config";
  var xmlHttp = new XMLHttpRequest();
  xmlHttp.onreadystatechange = function() {
    if(xmlHttp.readyState == 4)
      onConfig(getJsonResponse(xmlHttp.status, xmlHttp.responseText));
    }
 xmlHttp.open("GET", url, true); // true for asynchronous
 xmlHttp.send(null);
 }

The code above shows how the current configuration is read from the server and populate the form fields. If there is an error in the JSON data (or we didn't receive JSON in response) it simply hides the form and displays an error message instead. The 'showSection()' function used in that call is used to chose which of the main 'div' tags in the page to show. Saving the configuration is similar - the 'saveChanges()' function (shown below) is triggered when the 'Save' button is pressed. This collects the current values of the form fields, builds a JSON string and sends it to the server. If this results in an error we simply show an error message and block, if it succeeds we assume the device will reset itself with the new values and join the network.

function saveChanges() {  
  // Get the config from the form
  var result = { }
  result["ssid"] = document.getElementById("ssid").value;
  result["password"] = document.getElementById("password").value;
  // Post it to the server
  var url = "http://" + location.hostname + (location.port ? ':' + location.port : '') + "/config";
  var xmlHttp = new XMLHttpRequest();
  xmlHttp.onreadystatechange = function() {
    if(xmlHttp.readyState == 4) {
      var result = getJsonResponse(xmlHttp.status, xmlHttp.responseText);
      if (!result["status"])
        showSection("error");
      else
        showSection("finished");
      }
    }
  xmlHttp.open("POST", url, true);
  xmlHttp.setRequestHeader("Content-type", "application/json");
  xmlHttp.send(JSON.stringify(result));
  }

By keeping everything in a single HTML document we can serve it on a single URI without having to worry about additional requests for images or JS files for the code. Because I wanted any URL to display the config page I serve this page using the 'onNotFound()' handler so the client doesn't have to explicitly request the index page. Setting this up with the ESP8266WebServer is relatively straight forward. The code below shows how it is done (this code also sets up the '/config' URL as well which I will discuss later).

ESP8266WebServer httpServer(80);

void handleNotFound() {  
  httpServer.send(404, "text/plain", "Resource not found.");
  }

void handleDefault() {  
  httpServer.send_P(200, PSTR("text/html"), CONFIG_PAGE);
  }

void setupWebServer(bool withForm) {  
  httpServer.on("/config", handleConfig);
  if(withForm) {
    httpServer.onNotFound(handleDefault);
    }
  else
    httpServer.onNotFound(handleNotFound);
  httpServer.begin();
  }

Embedding the content of the page into your code can be a little tricky - you could define it is a single large string and escape the special characters but this makes it a bit difficult to maintain. My approach was to keep the page as a separate HTML file and use a Python script to convert it into an array of bytes stored in PROGMEM as the CONFIG_PAGE variable in the code sample above. Serving the page then becomes very simple as you can see - simply tell the web server to serve the data stored in that variable. The script I used to do this is shown below - it simply generates a C++ file containing the definition of a character array with all the bytes from the source file. An additional 0x00 is appended to the end of the data to make sure it is a NUL terminated string.

import sys  
from os.path import splitext

if __name__ == "__main__":  
  if len(sys.argv) <> 2:
    print "You must specify a single filename on the command line."
    exit(1)
  # Read the data file
  with open(sys.argv[1], "r") as input:
    data = input.read()
  # Generate the output
  output = open(splitext(sys.argv[1])[0] + ".cpp", "w")
  output.write("const char CONFIG_PAGE[] PROGMEM = {\n  ")
  count = 0
  for ch in data:
    output.write("0x%02x, " % ord(ch))
    count = count + 1
    if (count == 12):
      count = 0
      output.write("\n  ")
  output.write("0x00\n  };\n")
  output.close()

The steps above set up the extra functionality needed in 'configuration' mode - all that remains is to set up the functionality that is available in both modes, the ability to read and write the current configuration and support for discovery. This is one place where I needed functionality that wasn't already built in to the ESP8266 libraries - I wanted the configuration to be transferred in JSON format which will make writing scripts to query and modify it a lot easier. Processing JSON in C is not a trivial task - C lacks the dynamic properties of scripting language so building up an arbitrary structure from JSON input is extremely difficult. Luckily I found the JSMN (pronounced 'Jasmine') library that provides a lower level approach to the task - after adapting this into a Arduino compatible library I was ready to go.

The configuration data is access through the '/config' URL - any request to this URL will return the current configuration in JSON format, sending a POST request with a JSON formatted body will update the configuration values. The implementation is shown in the code snippet below:

void handleConfig() {  
  bool status = false;
  if(httpServer.method() == HTTP_POST) {
    // Process the configuration
    bool status = false;
    if (httpServer.hasArg("plain")) {
      JsonToken tokens[8];
      JsonParser parser(tokens, 8);
      String json = httpServer.arg("plain");
      int count = parser.parse(json.c_str());
      if(count > 0) {
        // Extract the settings
        int field = parser.find(0, "ssid");
        if((field>0)&&(parser.str(field)!=NULL)&&(parser.len(field)<MAX_SSID_LENGTH)) {
          memset(Config.m_szSSID, 0, MAX_SSID_LENGTH);
          strncpy(Config.m_szSSID, parser.str(field), parser.len(field));
          }
        field = parser.find(0, "password");
        if((field>0)&&(parser.str(field)!=NULL)&&(parser.len(field)<MAX_PASSWORD_LENGTH)) {
          memset(Config.m_szPass, 0, MAX_PASSWORD_LENGTH);
          strncpy(Config.m_szPass, parser.str(field), parser.len(field));
          }
        }
      }
    }
  // Build the response with the current values
  String response = "{\"ssid\": \"";
  response += Config.m_szSSID;
  if (httpServer.method() == HTTP_POST) {
    response += ", \"status\":";
    response += (status ? "true" : "false");
    }
  response += "\" }";
  httpServer.send(200, "application/json", response);
  }

One of the problems I originally had was accessing the raw JSON data in the POST request, the ESP8266WebServer has direct support for query strings (URLs with parameters encoded in the URL like '/config?ssid="My SSID"') and form data but it wasn't clear how to access data sent in other formats. Looking at the code I found that non-form data was being added as an argument called 'plain' and accessed as if it were a normal form field.

Putting it Together

The final step is to combine all of this into a library that can be included in a project. I created a library call 'IotConfig' that wraps everything in a singleton class and tried to simplify the interface to it as much as possible. This is still a work in progress so some of the finer details still need to be sorted out - my current test sketch looks like the following:

#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>
#include <DNSServer.h>
#include <Json.h>
#include <IotConfig.h>

bool running;  
void setup() {  
  Serial.begin(115200);
  running = IotConfig.setup(false);
  }

void loop() {  
  IotConfig.loop();
  if(!running)
    return;
  // Your sketch here
  }

As you can see using the library is relatively straight forward - a single 'setup()' method to initialise the library and a 'loop()' method to handle all the background operations that need to be done to support network transfers. The 'setup()' method supports a single boolean parameter that you can use to force configuration mode - to support a 'factory reset' operation for example, hold a button down on power up to clear the current configuration and start from scratch. The method returns 'true' if the device is ready to run, 'false' if it has gone into configuration mode - this value should be used to determine if your custom sketch should be run or not.

There is also a global 'Config' structure defined that contains the current configuration, after the call to 'IotConfig.setup()' this will be populated with the current configuration values that the rest of the sketch can use. All in all it is a pretty easy way to add support for configuration.

Next Steps

As I mentioned earlier this is still a work in progress so it will be a little while before I publish the source code. There is a fair bit of left over code from earlier experiments that I need to get rid of or clean up for a start. On top of that there are a few enhancements I would like to make:

  • Rather than force the user to type in a network I could present a list of networks visible to the device and allow the user to select one of those.
  • Add support for additional configuration parameters - all of my sensors communicate over MQTT so the address of the MQTT server and the topic to communicate on are obvious candidates.
  • Add an event mechanism to allow the user sketch to respond to configuration changes when they happen while in normal running mode. Changes to the network configuration will require a restart to apply but changes to MQTT should be able to be handled dynamically without restarting.
  • To save some space it might be possible to compress the HTML page using gzip and serve it as a compressed page using the 'Content-Encoding: gzip' header.
  • Security is an issue as well - at the moment the WiFi password is being sent in clear text over an unsecured network which is not very desirable. The password is never sent back with the rest of the configuration so the risk is partially limited but it would still be nice to make it as difficult as possible to see it.

Hopefully I will be able to finish off the library and address at least some of the improvements above over the next week or two. I will push the code to GitHub as soon as that's done and start integrating it into some 'real world' sensors. Stay tuned.