Paper forms, irrigation technicians, and a disk that filled up

My work at Personitech was mostly workflow digitization. Our customers’ workflows depended on paper forms, and I moved those workflows onto devices.

What I built

The platform is called ProcessSmart. The API was Java, deployed as WAR files onto Tomcat 8.5. I wrote the endpoints from scratch against JAX-RS, with no Spring or other framework. It ran on EC2. I designed and implemented the whole MySQL schema, which moved to RDS partway through. Photographs went to S3, and I archived media older than three months in Glacier.

The web client was an Angular 6 and TypeScript app. The Android client was native Java and authenticated against the API with tokens. Every page in both apps was gated behind authentication and authorization.

I was Personitech’s only engineer and built almost everything myself. The exceptions were the UI polish, which I had help with, and some backend code for West Highlands Pacific, which students I mentored wrote.

Working without a signal

Irrigation technicians were one of our main users, and they went to places with no data coverage. The use case required local storage, so the Android app kept its data in SQLite on the phone. When the technician had a connection again, an upload button sent the data to the cloud.

What the forms were for

Janitors, irrigation managers, and project managers submitted forms. An office manager read the submitted forms in the web app and approved or rejected them.

The greenhouse

Personitech partnered with a Taiwanese hardware team whose IoT gateway was deployed on a greenhouse in San Diego. The gateway transmitted data over MQTT, and I was in charge of the web dashboard that displayed charts and data from that stream.

The outage

The API started crashing, and at first I had no idea why. The first thing I did was set up a cron job that pinged the server every thirty seconds and restarted it when it didn’t answer. A restart took about a second. Before that, an outage usually lasted up to thirty minutes, and once my client notified me of an outage while I was away from the server, which was embarrassing. With the cron job, an outage lasted about thirty seconds. It was only a temporary fix while I looked for the root cause.

The restarts kept coming more and more often, so I suspected something on the server was bloating. I set up log files, and they showed the cause: the server was creating temporary files that quickly filled the storage on the free t2.micro it ran on.

To fix it, I moved the media files to S3 and upgraded to a larger instance.

What I changed after that, and what it cost

I split the deployment. Each client’s web app moved to its own instance, and the MySQL database moved to RDS after a coffee chat with a tech consultant, who warned me the database was a single point of failure. He later became my mentor.

Later, a denial-of-service attack hit my web API. I recovered, then started an IP blacklist to block traffic from a few countries where we had no customers. Our customers were in the United States and Taiwan.

The split more than tripled the billing from AWS. On the free tier the bill had been almost nothing. In September 2019 I moved from the free t2.micro to a t2.small, because of the crashes and the full disk, and that month’s bill was $18.99. The RDS free trial ran out in February 2020, and that month’s bill was $39.87. In October 2020 it was $45.31: $21.91 for EC2, $19.13 for RDS, $0.78 for S3, and $3.49 for data transfer.

Next time I’d probably keep MySQL on an EC2 instance, with daily syncs to my own computer, and keep S3. Each client would have their own EC2 instance for the web app and their own S3 bucket, and the API would run on a separate Personitech-owned instance. That would also split the billing properly between clients, which was another problem I had when I tried to scale to more customers.

Mentoring the IoT club

While I was at Personitech, I taught about six middle and high schoolers who had formed an IoT club at Canyon Crest Academy. I showed them what Personitech was doing with IoT. Each week we built one feature that touched the API, the frontend, and the database. I split the tasks by what each student was interested in, combined their work, and showed it to them at our weekly meetup.

The club then took on a project from the West Highlands Pacific Homeowners Association, a local homeowners’ association near the school: a web app and backend to digitally store the equipment and inventory of its community recreation center and to manage the renewal of that equipment, plus a form where guests could quickly log a visit. The app ran on ProcessSmart, and the students wrote some of its backend code. The team presented and demoed it to the association’s board in the recreation center’s meeting room, and the board approved it.

The students went from no app-building experience, with only basic computer science classes at school, to implementing a real application that’s useful to their local community.

What I couldn’t do

The iPhone and iPad app I built was a WebView linked to the web app. I didn’t know Swift or Objective-C, and I didn’t have time to learn them.