Paper forms, irrigation technicians, and a disk that filled up

Our customers ran their business on paper forms. The job was to move that onto devices without losing the parts of paper that actually work — chiefly that a sheet of paper does not need a signal.

What I built

The platform is called ProcessSmart. The API was Java, deployed as WAR files onto Tomcat 8.5, with the endpoints written against JAX-RS by hand — no Spring, no framework, written from scratch. It ran on EC2. I designed and implemented the whole MySQL schema, which moved to RDS partway through. Photographs went to S3, and anything older than three months aged into Glacier.

The clients were an Angular 6 and TypeScript web app, a native Java Android app authenticating against the API with tokens, and page-level authorization on both.

The part I would defend in an interview

Our heaviest users worked for an irrigation company — technicians, mostly. They drive out to places with no coverage, do a day of work, and come back. An app that needs the network to record anything is useless to them.

So the handset kept its own store — SQLite on the device, the full working set, everything recorded locally as it happened — and uploaded when the technician had a connection again, on a button they press rather than silently in the background.

What the forms were for

Capture was half of it. The other half was that a submitted form had to reach somebody with the authority to act on it. Janitors, irrigation managers and project managers filed from the field; an office manager sat in the office and read, approved or rejected what came in. Roles all the way through, with the permissions attached to the role rather than the person — which is why page-level authorization shows up in the list above rather than as an afterthought.

Paper does this with a physical in-tray and someone’s signature. The whole job was to keep that legible when the in-tray is a database and the signature is a row.

The greenhouse

Personitech partnered with a Taiwanese hardware team whose IoT gateway was deployed on a greenhouse in San Diego, publishing sensor readings over MQTT. I built the dashboard that consumed that stream and turned it into charts somebody could act on.

It is a small thing next to the rest of the platform, and it is the first system I worked on where the data arrived whether anything was ready for it or not.

The outage

The API started failing for no reason anyone could see. Restarting fixed it, then restarting stopped fixing it.

I had no logs. That was the actual problem, and the first real fix was writing somewhere to look — after which the answer took very little time: the server was generating temporary files, the free t2.micro it lived on had a small disk, and the disk was full. A full disk does not announce itself.

Two things came out of that, and they are not the same kind of thing.

The band-aid: a cron job pinged the server every thirty seconds and restarted it when it did not answer. A restart took about a second. Before it, an outage lasted until I noticed — up to half an hour, and once until my client phoned to tell me his software was down while I was away from it, which was embarrassing. After it, about thirty seconds. It was a band-aid and I knew it while I was writing it; it just bought me the room to find the cause.

The fix: media moved off the instance to S3, and the instance itself moved up a size. The thing that had been filling the disk stopped living on the disk.

What I changed after that, and what it cost

I split the deployment. Each client’s web app moved to its own instance, and the database moved to RDS — that one after a coffee with a consultant who pointed out that everything I had was sitting on a single box, and that I would find out the hard way. He was right, and he later became my mentor.

We were also hit by a denial-of-service attempt not long after. We recovered, and I added blocking at the edge for traffic from regions where we had no customers — ours were in the United States and Taiwan, and the traffic was not.

Here is the part I would want you to ask about: it more than tripled the AWS bill, and I do not think the trade was worth it. Small numbers — we went from about $17 a month on a single t2.small to nearly $60 once RDS, S3 and a second instance were in. Small enough that nobody made me justify it, which is exactly why I did not justify it carefully enough. Next time I would probably keep MySQL on an instance with a daily sync I held myself. I would keep S3, keep a web app instance per client, put the API on its own, and give each client their own bucket — which also fixes what I never solved: I could not tell you what any one customer cost us to run.

The app my students shipped

While I was there I taught a group of six students — middle and high schoolers who had started an IoT club at a nearby high school — what we were building. We ran it as a weekly build rather than a lecture: one feature a week, touching the API, the frontend and the database, with the pieces split by what each of them was actually interested in and integrated before the next meeting.

They then took on a real client. A local homeowners’ association wanted their community recreation centre’s equipment and inventory recorded properly, along with the schedule for replacing it, plus a fast way for guests to log a visit. The students built it on ProcessSmart — another workflow moving off paper, the same as the rest of our customers — and presented it to the association’s board in the recreation centre’s own meeting room. The board approved it.

They arrived with a school computer science class between them and no experience building anything real. The community is West Highlands Pacific; the club was at Canyon Crest Academy.

What I could not do

The iPhone and iPad app was a WebView wrapped around the web app. I did not know Swift or Objective-C, and there was no version of that year where I had time to learn them. It worked and our iPad users were served.