SoC-based distributed control system.

Russian-language version here.

Introduction.

In a complex distributed control system, many different tasks have to be solved on different levels.

Some tasks are better addressed at the level of the embedded PC with a full OS. Full fledged operating systems are convenient for them because of many useful tools, such as multithreading, ready-made drivers, libraries, different frameworks, and so on. And all this can be developed in high-level languages, especially without going into details of implementation at the lower level.

Some tasks are more convenient to solve at the level of the microcontroller (MCU), even without OS (bare-metal) or with minimalistic real-time OS. In this case, perhaps, key feature is the possibility of debugging software inside the OS by JTAG and monitoring what is going on at the periphery of the MCU at any break-point.

And there are tasks that should be solved at the FPGA level, since a microcontroller may not suit for a proper parallel control of different high-frequency electronics, for example, step motor drivers with encoders and speed regulators. In such tasks, the processor might be simply superfluous.

The number of executive devices and their various functions in the control system are significally increased when it comes to developing the device, for example, with a three-coordinate manipulator, a couple of servo motors, a dozen discrete devices, a bunch of peripherals on all popular interfaces (SPI, I2C, UART and other) and complex logic with mathematical analysis inside. And it would be very convenient to arrange the entire control system on a single chip. As a result, all three levels of control of PC-MCU-FPGA and their interactions are moved inside the chip.

In this case, inevitably arises the problem of creating a transport level that could connect all this complex logic with each other. For the MCU-FPGA link, the solution is, in fact, in creating peripheral device on the common MCU bus with its own set of registers. But the task of creating the transport level for PC-MCU has to be solved in a slightly different way.

The implementation of this bundle based on Altera Cyclone V SoC using Avalon Mailbox is described in this article.

For experiments we need to have a real or virtual machine with Ubuntu 16.04 on board.

Open-source code is available on GitHub.

Control system architecture.

Let's represent all executive devices of the PC-MCU-FPGA control system as parallel I/O ports. For example, meaning simple sensors and actuators, we can limit ourselves to a set of buttons and LEDs, and then manage them from the terminal command line.

All elements of FPGA, including MCU, are synthesizable. Part of the PC is already integrated into the chip and is based on the Cortex A9, whose buses are connected to the FPGA and can be used directly. Therefore, all that needs to be done is to connect communication modules of the synthesized nodes inside FPGA to the core of the OS via standard means.

DE0-Nano-SoC kit is used as a hardware platform for experiments.

FPGA firmware.

Demo project my_first_hps-fpga_base from the set DE0-Nano-SoC CD-ROM (rev.B0 / rev.C0 Board) will be used as a basis for further development. This project contains a preconfigured environment with properly installed FPGA ports, a ready-made Cyclone V Hard Processor System unit with the configured memory parameters and a set of auxiliary elements in Qsys. To work with the project, we need Quartus Prime 15.1 with Cyclone V support package and SoC Embedded Design Suite.

Some changes need to be made to the project. Add a NIOS kernel, a memory for it (16 KB of 32-bit width) and a JTAG port. In the NIOS parameters, select the addresses of vectors from the added memory.

Avalon Mailbox is simplex, so we need two modules (like the RX and TX lines of a conventional UART). The interrupt signal of each of the modules must be connected to the processor for which the module is a receiver.

Add one input port and one output port (8 bits each) for further testing of the system.

After adding all the elements, automatically select addresses and interrupts.

Create ports for buttons and LEDs in the source code of the top module.

  // Ports
  wire [7:0] port_out;
  assign LED = port_out;
  wire [7:0] port_in;
  assign port_in = {{2{1'b0}}, SW, KEY};

Connect the ports to soc_system.

  // FPGA Partion
  .port_out_export(port_out),  // port_out.export
  .port_in_export(port_in),    // port_in.export

Build the project and obtain the FPGA firmware, which will be a basis for further development.

Algorithm.

Let's create a system that does following:

• Toggle switch activates the timer;
• Each of the LEDs lights up in order with a frequency of 1 Hz by timer;
• Pressing the button changes direction;
• When the READ command is received, the current active LED number is sent from the PC to the standard Linux console;
• WRITE command from the PC changes current active LED;
• When the REVERSE command is received from the PC, the direction changes, as well as from the button;
• When another button is pressed, the counter of changing of LEDs starting from the last change of direction is sent to the PC console.

On the side of the MCU.

In the environment NIOS II EDS, which is essentially Eclipse with all the necessary plug-ins, create a new project soc_nios from the template "NIOS II Application and BSP". The result is two projects: the firmware itself and BSP.

First of all, you need to assemble BSP, but not in the traditional way. Instead, select BSP Editor from the NIOS II menu in the soc_nios_bsp project's context menu and turn on enable_small_c_library and enable_reduced_device_drivers to reduce target firmware binary size. Then compile by clicking Generate. In the future, since the assembly parameters are preserved, BSP can be simply rebuilt by selecting NIOS II from the Generate BSP menu item.

In the system.h file from the BSP project, you can see all the parameters of the MCU periphery that were previously added in the Qsys schema.

More details about NIOS and how to build NIOS projects you can read here.

For the implementation of MCU part of the algorithm we need:

• Timer interrupt handler;

  void TIMER_0_ISR(void* context){
    IOWR_ALTERA_AVALON_TIMER_STATUS(TIMER_0_BASE, 0);
    IOWR_ALTERA_AVALON_TIMER_CONTROL(TIMER_0_BASE, ALTERA_AVALON_TIMER_CONTROL_CONT_MSK);
    led += step;
    if(led > LED_MAX)
    {
      led = 0;
    }
    if(led < 0)
    {
      led = LED_MAX;
    }
    IOWR_ALTERA_AVALON_PIO_DATA(PORT_OUT_0_BASE, (1 << led));
    count++;
    IOWR_ALTERA_AVALON_TIMER_CONTROL(TIMER_0_BASE, ALTERA_AVALON_TIMER_CONTROL_CONT_MSK | ALTERA_AVALON_TIMER_CONTROL_ITO_MSK);
  }

• Mailbox interrupt handler;

  void MAILBOX_HPS2NIOS_ISR(void* context){
    IOWR_ALTERA_AVALON_MAILBOX_INTR(MAILBOX_SIMPLE_HPS2NIOS_BASE, 0);
    //NOTE: Order is important! CMD register should be read after PTR register
    buffer[1] = IORD_ALTERA_AVALON_MAILBOX_PTR(MAILBOX_SIMPLE_HPS2NIOS_BASE);
    buffer[0] = IORD_ALTERA_AVALON_MAILBOX_CMD(MAILBOX_SIMPLE_HPS2NIOS_BASE);
    alt_printf("Reading: 0x%x 0x%x\n\r", buffer[0], buffer[1]);
    newMail = true;
    IOWR_ALTERA_AVALON_MAILBOX_INTR(MAILBOX_SIMPLE_HPS2NIOS_BASE, ALTERA_AVALON_MAILBOX_SIMPLE_INTR_PEN_MSK);
  }

• The message parser and the Mailbox write function;
• Button polling and LED control functions.

It remains only to assemble the project. The size of NIOS firmware should be less than 16 KB.

To test the firmware on real hardware, a new debugger configuration needs to be created. After flashing the FPGA from the Quartus Programmer in the Debug Configurations menu, select the NIOS II Hardware option, update all the interfaces, and find jtaguart_1 in the Target Connections tab. This is exactly that JTAG for NIOS, which was previously added to Qsys especially for debug.

Now debugging from Eclipse can be started. If everything was done correctly, the message "Turn the switch ON to activate the timer" should appear in the NIOS II console.

On the side of the PC.

Installing Linux on the board.

The entire process is described in detail here in sections 1 to 10. It is recommended to use more recent versions of toolchain, bootloader and kernel, than those that can be found in that manual. Note that building this version of the bootloader does not work with a compiler above the 6th version.

To generate the device tree, instead of the proposed sopc2dts utility, it's better to use the sopc2dts.jar script with parameter --type dtb.

It is strongly recommended to use the latest Buildroot. For building the system, CC environment variables need to be specified as the path to arm-linux-gnueabihf-gcc and CXX as the path to arm-linux-gnueabihf-g++ from the toolchain. Next, enter the currently used versions of the compiler, the kernel, and the library (the system suggests them itself during the build process). In the settings of the toolbox in the Buildroot configuration, you must always specify the path to the toolchain, as well as the $(ARCH)-linux-gnueabihf prefix and enable SSP, RPC and C++ support.

Packages nano, mc and openssh can be added to Buildroot for convenience.

Next, all top-level software will be built in Eclipse with the GNU MCU Eclipse plug-in. Create a new workspace for ARM projects and in the global Eclipse settings in the Workspace Tools Path section, specify the corresponding path to the installed version of Linaro.

Driver.

First of all, let's make the driver for Mailboxes. Create a new nios_mailbox project in Eclipse from the "Hello World ARM C Project" template.

In the project settings, turn off the options "Use default build command" and "Generate Makefiles automatically", since the command make TARGET=nios_mailbox TARGET_DIR=Default is required to build the kernel module. Add to the environment variables two new entries CROSS_COMPILE and KDIR, indicating the full path with the toolchain prefix and the path to the source code of the kernel. Define __GNUC__, __KERNEL__ and MODULE in the correspondind list. That's it, now it's time to write the code.

The kernel module will respond to the interrupt from the hardware and should somehow report this to the application world. For this purpose a new user signal needs to be created.

  #define NIOS_MAILBOX_REALTIME_SIGNO  44

The driver will be created by using a platform_device type, each Mailbox will be as miscdevice, and eventually will be visible as a device file in the /dev directory. More details about Linux drivers and system in general can be found here. It is important to understand that there can be many Mailboxes, but the driver is one for them all, and they should all be initialized and numbered in right way.

To put it simply, the development of the driver actually comes down to the implementation of standard read and write operations with a file, plus a small bonus function ioctl(), which is needed for telling driver an id of the process that uses it. It is necessary for a driver to know which process needs to be awared by signal in case of the hardware interrupt. The interrupt handler itself looks quite simple and very similar to its counterpart in NIOS.

  static irq_handler_t nios_mailbox_isr(int irq, void *pdev)
  {
    struct nios_mailbox_dev *dev = (struct nios_mailbox_dev*)platform_get_drvdata(pdev);
    spin_lock(&dev->lock);
    //NOTE: Order is important! CMD register should be read after PTR register
    dev->data[1] = ioread32(dev->regs + ALTERA_AVALON_MAILBOX_SIMPLE_PTR_OFST * sizeof(u32));
    dev->data[0] = ioread32(dev->regs + ALTERA_AVALON_MAILBOX_SIMPLE_CMD_OFST * sizeof(u32));
    spin_unlock(&dev->lock);
    if(dev->task)
    {
      send_sig_info(dev->sinfo.si_signo, &dev->sinfo, dev->task);
    }
    return (irq_handler_t)IRQ_HANDLED;
  }

It remains to build the project. For this we need to write a special Makefile. It will look like this.

  all:
    @echo 'KDIR=$(KDIR)'
    @echo 'CROSS_COMPILE=$(CROSS_COMPILE)'
    @if [ ! -d $(CURDIR)/$(TARGET_DIR) ]; then mkdir $(CURDIR)/$(TARGET_DIR); fi
    cp $(TARGET).c $(CURDIR)/$(TARGET_DIR)
    cp $(TARGET).h $(CURDIR)/$(TARGET_DIR)
    cp Kbuild $(CURDIR)/$(TARGET_DIR)
    $(MAKE) -C $(KDIR) ARCH=arm M=$(CURDIR)/$(TARGET_DIR)

  clean:
    rm -rf main $(CURDIR)/$(TARGET_DIR)

And we need to create a Kbuild file with one line.

  obj-m := $(TARGET).o

Build the project in the traditional way. The result is a kernel module nios_mailbox.ko, which needs to be copied on the system and installed using insmod. If done correctly, in the Linux console opened via USB, pressing the appropriate button on the board should lead to the appearance of a message from the kernel "[.........] NIOS Mailbox new mail!".

Of course, it would be very useful to add a buffer to the driver for the received data by the interrupt, since the reading in the application program can not keep up with the data flow from hardware. And it is better to compile the driver itself with the option stripped, to save space in the embedded system. However, these improvements are left to the reader for self-study.

Application.

So now we got to writing the console application. Create a new project in Eclipse soc_test from the "Hello World ARM C++ Project" template. In the Settings section of the Target Processor select the cortex-a9 architecture, in the Cross ARM GNU G++ Linker add -pthread. In the Build Artifact tab remove the file extension. All other settings can be left by default.

For the application part of the algorithm we need:

• The signal handler;

  void Nios::mailbox_nios2hps_signal_handler(int signo, siginfo_t *info, void *unused)
  {
    if(info->si_signo == NIOS_MAILBOX_REALTIME_SIGNO)
    {
      sem_post(&mailbox_nios2hps_signal_semaphore);
    }
  }

• Message parser from Mailbox;

void *Nios::mailbox_nios2hps_data_reader(void *args)
{
  uint64_t mes;

  while(1)
  {
    while(sem_wait(&mailbox_nios2hps_signal_semaphore));
    if(lseek(mailbox_nios2hps, 0, SEEK_SET) != 0)
    {
      cerr << "Failed to seek mailbox_nios2hps to proper location" << endl;
      continue;
    }
    read(mailbox_nios2hps, &mes, sizeof(mes));
    printf("[HARDWARE] Reading: 0x%08x 0x%08x\n", (uint32_t)mes, (uint32_t)(mes >> 32));
    switch ((uint32_t)mes) {
      case LED_NUMBER:
        printf("Active led %lu\n", (uint32_t)(mes >> 32));
        break;
      case SWITCH_COUNT:
        printf("Led switched %lu times\n", (uint32_t)(mes >> 32));
        break;
      default:
        break;
    }
  }
  return NULL;
}

• Messagebox write function;

void Nios::mailbox_hps2nios_write(uint64_t mes)
{
  if(lseek(mailbox_hps2nios, 0, SEEK_SET) != 0)
  {
    cerr << "Failed to seek mailbox_hps2nios to proper location" << endl;
  }
  else
  {
    printf("[HARDWARE] Writing: 0x%08x 0x%08x\n", (uint32_t)mes, (uint32_t)(mes >> 32));
    write(mailbox_hps2nios, &mes, sizeof(mes));
  }
}

• The setup procedure with device files which have appeared after the driver installation;

Nios::Nios ()
{
  struct sigaction backup_action;

  pid = getpid();

  mailbox_nios2hps = open("/dev/nios_mailbox_0", O_RDONLY);
  if(mailbox_nios2hps < 0)
  {
    cerr << "Could not open \"/dev/nios_mailbox_0\"..." << endl;
    exit(1);
  }
  memset(&mailbox_nios2hps_action, 0, sizeof(struct sigaction));
  mailbox_nios2hps_action.sa_sigaction = mailbox_nios2hps_signal_handler;
  mailbox_nios2hps_action.sa_flags = SA_SIGINFO | SA_NODEFER;
  sigaction(NIOS_MAILBOX_REALTIME_SIGNO, &mailbox_nios2hps_action, &backup_action);
  if(ioctl(mailbox_nios2hps, IOCTL_SET_PID, &pid))
  {
    cerr << "Failed IOCTL_SET_PID" << endl;
    close(mailbox_nios2hps);
    sigaction(NIOS_MAILBOX_REALTIME_SIGNO, &backup_action, NULL);
    exit(1);
  }

  mailbox_hps2nios = open("/dev/nios_mailbox_1", (O_WRONLY | O_SYNC));
  if(mailbox_hps2nios < 0)
  {
    cerr << "Could not open \"/dev/nios_mailbox_1\"..." << endl;
    close(mailbox_nios2hps);
    sigaction(NIOS_MAILBOX_REALTIME_SIGNO, &backup_action, NULL);
    exit(1);
  }

  pthread_create(&nios2hps_data_reader_thread, NULL, mailbox_nios2hps_data_reader, NULL);
}

• Parser of console commands.

It remains to build the project. The result is an executable file for the ARM-9 architecture that needs to be copied to the system. If everything is done correctly, the message "Enter command ("read"("r"), "write"("w"), "reverse"), "q" to exit" will appear in console.

Running and testing the system.

Kernel module installation is added to the Linux startup.

Build a new version of NIOS firmware without all JTAG debug output. Convert the firmware to hex format by running inside SoC EDS 15.1 Command Shell command "elf2hex --input=soc_nios.elf --output=soc_nios.hex --width=32 --base=0x4000 --end=0x7fff --record=4". The resulting firmware should be added as an initialization file for NIOS memory in Qsys, then rebuild Qsys, rebuild the FPGA project and write a new firmware to the memory card.

Download and run the test application at once. And if everything was done correctly, the system should run as it supposed to according to the algorithm.

Summary.

Do not hesitate to use such complex bundles as a FPGA-MCU-PC based on SoC in embedded projects. This article demonstrates that it is not so difficult to implement such a system, as it might seem in the beggining. You can even add several microcontrollers and link them together in the same way.

The control system, created on the basis of the principles set out above, was introduced by the author in one of the electronic devices and has proven its performance in the real world.