Randomness

One of the things I like best about being a software engineer is that I get to embed myself deeply into other industries. I believe you have to have empathy for the users of your software; that you have to put yourself into the mindset of the people making the industry work. Sometimes the specific industry you work in is a choice, or a calling, but it can also be as random as a recruiter reaching out on LinkedIn asking if you've ever thought about moving to New York City. This randomness is what inspired the move I made in 2015; I didn't know it at the time, but I would spend the next 11 years (so far) in the finance industry.

At this point in my career I had worked in a variety of industries: telecommunications, food technology, healthcare; each with their own unique challenges, joysmy favorite joy of the food industry was visiting the Dunkin Brands headquarters in Canton, MA and sampling their experimental ice cream and doughnut flavors 🤤, and stakes. My first role in the finance industry turned out to be working on an investment guidelines surveillance engine at an asset management firm. To translate for my non-finance friends: we were the final guardrail protecting our customers and the firm from a "bad trade" (a trade that may violate one of the myriad rules governing portfolio balances, restricted securities, or any other criteria our operations users entered into the system). It was an eye-opening shift in the standards of correctness and operational reliability required to keep things going well. Because when things stopped going well, that's when I'd get the much-dreaded support call.

The much-dreaded support call

Anyone who works in the software business knows them well. We were supporting a system that people relied on to do their jobs. When the rubber of software meets the road of reality, the treads wear down and eventually burst to reveal a flaw that's been waiting to be unearthed. If we were lucky, we caught the flaw before it became a widespread issue. If we were unlucky, I'd receive an angry call from someone. Suddenly our system (and by extension, me) was preventing them from doing their time-critical job, from executing a timely portfolio rebalance. The firm's reputation (and money) was on the line.

So, with the trader breathing down my neck (sometimes literally), I got to work diagnosing. Sweatily pulling data from the trading system, and from the surveillance engine, and then carefully walking the rules down their various paths and branches, and diving deep into the calculations underlying the rule's logic until I found the root cause of the discrepancy. It took time when there was no real time to be given.

Each call ended the same way: trade unblocked, caller placated, root cause still in place. A system that has the power to prevent trades owes its users a defensible answer every time, whether or not anyone calls to complain. So I decided to treat the incidents as data rather than interruptions: what patterns kept appearing, and how could we prevent the whole class of problem instead of the individual complaint? (That this also promised a saner on-call existence was a happy bonus.)

Over my first several months on the job I realized there was a common thread weaving the incidents together: it's not that there's a runtime error or an NPE; it's simply that a rule has tripped unexpectedly and prevented a trade from executing. Surprisingly, in most of these cases nothing was even "broken"in the traditional, "oh no! this code has a bug", sense of the word. The desk's system had computed the position as liquid and ours decided it was illiquid (or similar). Both were "working as designed" and as far as these systems knew, they were just reporting their own truth. They simply disagreed about the very definition of reality liquidity itself. Confusing? Yes.

Disagreeing about reality

This is one of those moments where a problem shifts from one class of problem to something else entirely. Why would two systems, built by two competent teams, disagree about something as fundamental as liquidityLiquidity is the concept of how easy it is to buy or sell something for cash. Cash is the most liquid because... it's cash! Most publicly traded stock is fairly liquid and you wouldn't struggle to find a buyer/seller under normal circumstances (for the right price). Other assets behave differently and since there are regulations that require certain ratios of a portfolio being "liquid" these definitions matter.?

The problem was that liquidity wasn't a single formula. Our definition was actually built from several dependent calculations over many data points. Some of these were dependent on credit ratings. And credit ratings introduce two subtle sources of divergence on their own. Different ratings agencies rate the same instrument on different letter scales, so before we could use ratings in a calculation we had to normalize them onto one common scale, and each team had (naturally) implemented their own normalization calculation. Not to mention that the ratings data itself could be sourced from different reference databases, with slightly different formats and values.

Every layer is another opportunity for two implementations to drift apart!

This is when I started to realize that what I really needed to do was to convince everyone to use the same data model and define calculations the same way! For everything! Definitely easier said than done; it was clearly a problem above my pay-grade, but I knew what needed to be done so I began to ask around.

I read other teams' code and talked with their engineers, then with desk heads, portfolio managers, traders. The same pattern was everywhere: shared calculations, but with divergent definitions. And the people consuming these numbers had their own version of the complaint: a liquidity score taken by itself is very abstract. There was no easy answer to "where does this number come from?" that didn't involve having an engineer go back to read the source code.

The more I interviewed the different stakeholders across the company, the more the picture of what we needed took shape. Wouldn't it be great if we had a way to not only unify our calculation definitions, but to also get automatically updating documentation "for free" as well? What if anyone at the company could search in a UI for the calculations and debug them step by step for any given instrument/position/etc? Wouldn't it also be great if our business users who thoroughly understood these calculations could simply define the calculations themselves?

Calculations as graphs

As I began reflecting on the problem and diagramming calculation definitions, I realized that all of these calculations could be described in a tree structure! After all, isn't that how I modeled them in documentation? So, I decided to start modeling the calculations as graphs.

More specifically, I designed a model that allowed me to define each calculation as an abstract syntax tree (AST). The leaves are either fields from a shared data model or constants; every other node is an operation applied to its inputs: arithmetic, comparisons, conditional logic, aggregations over lists. It's a small, recursive structure, and it can express essentially any calculation we needed.

The built-in "side-effect" of modeling calculations this way is that it could not only be evaluated to provide a value, but it could also explain to an end user, step by step, how that value was arrived at. A calculation that is a graph can be displayed as easily as calculated. It could give business users the same tools as engineers without needing to read the code. And because every leaf names a field, answering "what depends on this field?" stops being an archaeology project and becomes a query: a simple tree traversal. But first, I needed to prove the basics.

Let's call these computed attributes. I deliberately started with a Java API. Our theory of adoption: engineers don't adopt mandates; they adopt good libraries. Give every team one well-designed way to define calculations over plain objects from a shared data model, with no runtime dependency on any central service, and the definitions come along for free.Agreeing on that shared data model took sustained negotiation across teams and wasn't fully realized in my time there; easily the hardest non-engineering work of the whole project.

Declaring an attribute in code looked roughly like this: name it with a slug, then compose it from typed field references. field() takes a method reference, so the compiler checks every field access against the shared data model; a mistyped field is a build failure. The result evaluates directly, strongly typed end to end.

import static attributes.Inputs.field;

// engineers declare attributes directly in code
ComputedAttribute<Position, BigDecimal> notional =
    ComputedAttribute.of(Position.class, "notional")
        .mult(field(Position::price), field(Position::quantity));

BigDecimal value = notional.evaluate(position);  // strong typing in the Java API

Attributes authored in the UI travel a different path to the same place. They live in the versioned store, and a consuming service fetches one by its unique ID, with the ID arriving through application configuration rather than code. From there it behaves identically to the declared kind: same tree underneath, same typed evaluate call.

// UI-authored attributes are fetched by unique ID, injected via app config
ComputedAttribute<Position, BigDecimal> liquidity =
    attributes.fetch(config.get("surveillance.liquidity-attr-id"));

BigDecimal score = liquidity.evaluate(position);

What about off-the-shelf rules engines? They existed, and we evaluated them. Two requirements ruled them out. The API was the adoption strategy, so we needed to control it end to end; and we wanted full control over evaluation, because I already suspected performance would matter later.foreshadowing Not to mention, Drools was the biggest library in this space (and still kind of is). Once I had the nice API, I didn't want to spend time code-generating Drools DSL. And with a good clean API serving as an abstraction layer, we could always change the evaluation engine behind the scenes one day if that proved best.

The pitch

With the help of my mentors, I brought the problem and the proposed shape of the solution to the executive running our division and asked for time to prove it out. To my surprise, he said yes! They gave me roughly a month, solo, to build a proof of concept. That's when I got to work on discovery, API design, discussions, and testing.

Presenting this POC to the executive and his senior leadership team became one of the more memorable meetings of my career: we all gathered around a conference room table where the senior-most technology group in our company spent an hour grilling me about the problem and the proposed solution, looking for architectural issues or any risk at all that this project might pose. One area they pressed me on particularly was audit: if business users can define calculations that gate trades, those definitions need version control, review, and a complete audit trail. That exact requirement had already surfaced in my stakeholder interviews, so the design had an answer ready. The project was approved.

What we ended up building may be easier to show than to describe.

What we built

Below is a reconstruction of the system's core, built from memory for this write-up: the tree editor, the type system, the definition library, versioning. Everything evaluates live against sample records.

✦ Click here if you want the guided tour.

Definitions are code

As seen on the "History" tab in the demo above, calculations defined in the UI were saved in immutable versions for audit reasons:

• Every save created a new version; nothing was ever edited in place.
• Versions started as drafts and had to be published to take effect; consumers never saw work in progress.
• Definitions promoted through environments the way code does, and the approver had to be someone other than the author.
• Calculations carried test suites: expected outputs pinned against known inputs, run on every change.
• All of it left an audit trail: who saved what, who approved it, who published it, when.

Another benefit of carefully versioning attributes in this way was that it preserved data lineage: before changing a field's meaning, we could enumerate every calculation that referenced it before touching production code.

Being correct and auditable, though nice, isn't everything. We also needed the calculation evaluations to be fast enough to be used in practical settings.

But is it fast?

The first evaluator was a straightforward recursive interpreter in Java: for each node, either read the leaf value from data or evaluate the children and apply the operator. Simple, correct, easy to reason about. Also a little slow: I recall a benchmark of roughly 200 milliseconds to evaluate one large-ish computed attribute against one input item. Portfolios vary in size, but with many customers, each with varied (and some very large) portfolios, we needed to do better.

I thought about the kinds of optimizations I could implement: caching, dead-branch elimination and so on. At some point I noticed that my list was kind of a description of standard compiler work, which made me wonder: what if I stopped interpreting and just compiled? Was it possible to generate Java source code from the AST, compile it in memory inside the running process, load it through an in-memory classloader, and invoke it like any other class? After much trial and error, it worked! Evaluation on the benchmark dropped from roughly 200ms to ~6ms for the same work.

Distribution fell out of the same design. A consuming service pins the ID of a computed attribute, hydrates the compiled class at startup, and refreshes whenever a new revision is published. Definitions changed infrequently, so polling (or simply restarting) was plenty; no pub/sub required. Teams kept their operational independence; definitions stayed centralized, versioned, and shared.

Fast, correct, auditable, and shareable. The remaining question was whether anyone would actually use it.

Retrospective

Shortly after the approval, the project stopped being a solo effort: four interns joined for a summer, then eight new analysts for an eight-week build-out, with me as the accountable engineer throughout. With the extra help, we added some great features like attribute deduplicationdetecting when you're defining an attribute that already exists by comparing the graph structures and the "preview" evaluation & debugger featureThis allowed users to search for a record to use as the input data; then they could see the evaluation and "debug" it by stepping into the graph as it's calculated (similar to debugging an Excel calculation)..

In the end, adoption of the "computed attributes" was narrower than I'd hoped. Sure, we built a working, complete product that met the needs of our users and provided real value in a production environment. But the initial goal was company-wide unification; in reality there was modest adoption within fixed income, where portfolio managers began creating their own computed attributes to classify instruments in their portfolios.

In retrospect, I can see that I made the case to engineers, and it worked on engineers. I talked to the stakeholders and built something they actually needed. But what I under-invested in was working with the people who set those engineers' roadmaps, to make sure capacity was set aside for adoption. The biggest lesson from this project is that solving big problems within a large organization is just as much a prioritization problem as an engineering one.