IoT Architecture & Development

"How do I get into IoT development?"

I've gotten this question from several people, and it's a long answer. Hence this article. Which was originally going to be one article, but I now realize needs to be a series of articles.

The first thing I want to question is "Do you really want to do IoT, or is there a particular aspect of IoT solution delivery that interests you?"

Some major areas I see this broken into:

• Enterprise & Application Architecture
• Wireless/Network Integration
• Security
• Data Collection & Warehousing (Data Architecture)
• Analytics/Machine Learning
• Electronics/Hardware Development & Integration
• Device Management

That's quite a list for anybody. You are probably not going to be expert in more than one or two areas. With over 20 years in technology development, I have several strengths on that list, but I definitely need to use expert consulting on at least half. Stuff that I am not going to talk about: Cloud (if you know anything, this is obvious and covered elsewhere) and blockchain (I am not an attention seeker, so I don't need to bullshit you - please use LinkedIn's search feature to find charlatans in that space).

Net/net: This is an overly ambitious series of posts that I am authoring by request. I hope it helps somebody. It may out me as an unqualified hack (but I don't think it will).

I am going to walk through my take on the areas above.

Coming from the angle that's most common, I'll only address two of these (in this first installment) which are familiar to the average developer -- Enterprise & Application Architecture and Data Warehousing. Most application developers get exposed to this (the twist is with the data collection -- and that is primarily because a lot of enterprise shops are doing batch ETL or relatively mundane data streams from a small number of endpoints to a single ETL platform).

Regarding the application: Recognize that any IoT project is supposed to solve for a business need. In theory, this is where the Enterprise people are supposed to thrive. Despite the traditional failure rates of software projects, Enterprise Architecture is intended to address real business needs with user-focused solutions. If you look through the list above, the only other touchpoint to users is Security. For the bulk of IoT projects, they will eventually boil down to some sort of user story (if you like the use-case methodology) or functional requirement. So that's an area to talk about on its own, even if it's not exotic.

It may seem spurious, but I'll repeat it: Any IoT project/system should boil down to a functional requirement that justifies the investment. Application developers will be part of the delivery, and if you don't believe that's a vital part of IoT, you are selling snake oil. Generic platforms (to aggregate data) and undifferentiated offerings (Tableau charts) do not a value proposition make. As such, I firmly believe a little application and enterprise development experience is important for an IoT team. Understand the vertical and take the time to create added value. If this doesn't make sense, spend some more time talking to your users and think about the difference between filling a database with rows and giving them a usable insight.

While application requirements are the simplest aspect of IoT, they are also the most important. User focus.

Regarding the data pipeline: IoT systems have thousands to millions of data sources. This is something that your average enterprise developer is not accustomed to, in either scale or complexity. As a result, this is one of the first challenge areas you're going to hit. And, when I started with replacing our IoT infrastructure, data collection was the first area I tackled (I have a strong ETL and data collection background, having worked in a content processing department ingesting data feeds from publishers, and also working on an ETL platform for advertising intelligence data -- relatively high scale and complexity for the time). You're going to be pushed to deliver a highly available, highly scalable, and highly secure data collection platform to feed the application database. And this means touch points to both the security and wireless/network integration side (there will be future discussion on this).

In our rebuild of our IoT platform, my boss pushed me to separate out ingest, storage, and ETL as discrete and independently scalable layers. At the time, I thought it was a bit over-engineered, but now I see the sense in it. Plan for at least 100x your current throughput from an architectural perspective, and consider the very real possibility you will need 10,000x. I have little doubt we'll be ingesting data from 100,000 sensors in the near future. Any worthwhile commercial IoT application will be in this range. Scalability is the name of the game, and you are concentrating many small pipes to a few big ones - so figure that out. Haven't thought about CoAP over UDP? I recommend you think about CoAP over UDP. Why? Because I recently had a talk with someone from one of the biggest IoT platform companies in the country who was looking at scalability of TCP-based concentrators.

Do you know how to handle 15,000 open file descriptors? Well, do you?

Next article, I'll talk about wireless and network integration.