Methods and Functions and Partials, Oh My!

(This week, I’m trying something new and saving code snippets in Scala Worksheets instead of Scastie. You can now run all my code samples locally by cloning the repo. You can also still copy code snippets in to Scastie too.)

My name is Brad and I’m a software engineer at Axon. Something that’s interesting about engineering at Axon is that we write most of our backend services in Scala. I did not know Scala (or functional programming generally) before starting here and had to pick it up along the way.

Today, I want to share a lesson that I learned recently about the different types of transformations that exist in Scala.

The ‘Function’ in Functional Programming

A basic difference between the imperative and functional programming paradigms is that, in imperative programming, we give the computer a sequence of commands that tell it what to do:

In functional programming, we instead start with some data and then repeatedly apply transformations to the data, yielding new data at each step:

If you’re new to functional programming, that may just seem like a more complicated method for arriving at the same outcome. I promise that there are benefits to this style (like immutability and referential transparency) but those aren’t the topic that I want to dive in to today. Rather, I want to focus on what’s happening between each of those map calls.

The building block of function programming is the transformation. A transformation is a rule for converting a thing into another thing.

Some of you may now be wondering why I’m calling it a transformation instead of a Function, and I promise there’s a good reason for that: Functions aren’t the only type of transformation in Scala. In fact:

Today, we’re going to be going over methods, Functions, and Partial Functions.

There is a Method to My Madness

Scala is a functional language, but it’s built on top of the JVM runtime which isn’t particularly functional. Rather, the JVM is first and foremost object oriented. All data has to belong to an object, and subroutines are data just like everything else.

The JVM method is essentially a wrapper that packages some bytecode together with some metadata which are then attached to a class. It’s a critically important primitive that allows us to organize and encapsulate functionality so that it can be reused.

In Scala, methods are created with the def keyword like this:

At the end of the day though, a method is only a primitive and it lives and dies by the class it’s attached to. Methods can’t stand on their own, be assigned to values, or be passed to or from other methods (or at least, not without some help). Ultimately, methods are a necessary but not quite sufficient ingredient for implementing functional programming.

Method 2.0: The Function

The functional programming paradigm requires a more robust type of transformation than what is provided by the humble method. To practice functional programming, we need to be able to pass functions to other functions as inputs and get additional functions back as output. Our transformations need to have types and not just signatures.

To build a functional language on top of a non-functional runtime, the Scala language designers had bridge the gap and they did it with the Function type. Function (and more specifically, Function0 - Function22) is a type that represent and encapsulates a method. It has a single instance method, apply, which is the method that we’re wrapping in a class. Let’s look at an example:

This abstraction is very powerful. We’ve taken a method and wrapped it in a class whose sole purpose is to represent that method. In doing so, we’ve made it into a first class citizen. It (or more accurately, the Function which wraps it) has a type. It can be saved to vals and passed to and from other Functions. It actually has instance methods of it’s own like compose and unlift. Effectively, we’ve taken a method and turned it into a full blown object.

In spite of all that power though, it still feels a little awkward. Being able to pass doubleIt to higher-order functions is cool, but between “new Function1[A, B]” and “doubleIt.apply“, that’s a decent amount of boilerplate code that I have to write to make it happen right?

The Scala language designers agree, and that’s why they gave us three pieces of syntactic sugar that make defining and working with Functions as clean as if they were just plain old methods:

First, the apply method is special in Scala because it doesn’t need to be named to be invoked. If we call the object itself as if it were a method, the compiler will dispatch that call to the apply method automatically.
Next, we can express Function types more cleanly with => notation. The type Function1[A, B] can also be written be written as A => B and the compiler will automatically expand it to Function1[A, B] for us. (This is also true for Function2 - Function22.)
Additionally, we can shorten our function definitions with lambdas. A lambda is an anonymous function. We can take an unnamed function and assign it to a named value, meaning we don’t need to explicitly define an apply method at all.

If we put all of these pieces together, we can greatly simplify our doubleIt Function:

This code is equivalent to the last example. The only change is that it’s now much more concise.

Eta Expansion? It’s All Greek To Me

Earlier, I said that methods don’t have types, can’t be assigned to values, and can’t be passed to higher-order functions, and that is still technically true.

But it isn’t entirely true:

So what’s going on here?

As with the syntactic sugar that makes Functions easier to define and invoke, the language designers also realized that there would be a frequent need to promote methods to fully formed Functions, and so they added a language feature to do that. This feature is called Eta Expansion, and under some circumstances, the compiler will use it behind the scenes to automatically stitch a program together.

In the above example, List.map expects a Function but was only provided with a method instead. Because map was expecting a Function1[Int, Int] and was given a method with the signature (x: Int): Int, the compiler automatically wrapped the method in a Function to make the program work.

Similarly, even though methods aren’t directly assignable to values, the compiler was able to make the statement val x = doubleIt work by implicitly wrapping doubleIt in a Function.

Overall, the Scala compiler is very good when it comes recognizing the need for Eta Expansions and automatically performing them when appropriate. If there’s every a circumstance where the compiler isn’t performing an automatic Eta Expansion for you, you can do it manually by putting an underscore after the method:

And there you have it. Because of a tremendous amount of work by the language designers, methods and Functions are interchangeable for nearly all intents and purposes.

Let’s Talk About Partial Functions

One of the hangups of Functions is that they have to be (more or less) complete. A normal function needs to be ready, willing, and able to accept and transform every member of it’s input type into something in its output type (or die trying).

Sometimes, this isn’t what we want thought. Not all functions are defined for all elements of their input type:

There’s another type of transformation which is better suited for this problem: The PartialFunction.

PartialFunction[A, B] is a subtype of Function1[A, B] (a.k.a., A => B). In addition to the apply method, PartialFunction also has the method isDefinedAt which determines if the given input is in the domain of the partial function:

Just like it’s parent class, PartialFunction is a powerful abstraction that’s saddled with some baggage in the form of awkward boilerplate code. Fortunately, the language designers have again given us some syntactic sugar to ease that burden too. The apply method can still be invoked implicitly and we can significantly shorten the definition of the PartialFunction by writing it with an Extractor (a.k.a. case statement):

The compiler will build a PartialFunction for us from the Extractor by converting the guard condition to the isDefinedAt method and the case body to the apply method.

Because PartialFunction[A, B] is a subclass of A => B and Extractors are implicitly convertible to partial functions, we can supply a case statement anywhere we need a PartialFunction and a case statement or PartialFunction anywhere we need a Function:

If you’re ok with exceptions in your calculations, this can be a quick and dirty way to verify preconditions. There are certainly cleaner ways of achieving the same result though, and this example should be interpreted more as a cautionary tale rather than as a how-to guide.

Do You Even Lift Bro?

Sometimes, we have a PartialFunction when it would be more convenient just to have a complete Function. Even though, as we just saw, partial functions are substitutable with complete functions, the strategy that any given PartialFunction uses for handling out-of-domain inputs may not be quite what we want, like how case statements throw exceptions by default. (Side note - partial functions don’t necessarily throw exceptions on out-of-domain inputs, but it’s common to do so. The behavior is ultimately determined by the implementation of the apply method.)

Fortunately for us, there’s actually a convenient path for converting partial functions to complete functions called “lifting”. In general, any PartialFunction[A, B] can be “lifted” into a Function1[A, Option[B]] where the result is a Some for in-domain inputs and None for out-of-domain inputs:

After we lift the partial function into a regular function, we can go on to handle the in-domain and out-of-domain cases using typical Option operations like map and getOrElse.

Additionally, it’s also possible to travel in the other direction too. If we have a “lifted” function (i.e. a function of the form A => Option[B]), we can unlift that down into a PartialFunction too:

There’s an important caveat to note here. Under some circumstances, the way that the new PartialFunction determines if the lifted function is defined at the input is by invoking it and testing if the result is defined. This means that the lifted function may actually be invoked twice for any logical invocation of the partial function. If the original function is making a database or web API call, that could be problematic and it may become necessary to do extra work to memoize your partial function.

Let’s Wrap Up

In this post, we learned about the three main types of transformations that exist in Scala and how they differ from one another:

Methods are JVM primitives that encapsulate bytecode for re-use. They have signatures but not types and can’t be saved to values or passed to or from other higher-order transformations (or at least not without help from eta expansion.)
Functions are an abstraction built on top of methods that bridge the gap between the abilities of methods and the needs of the functional programming paradigm. They are first-class values that can be assigned to values and passed to higher-order functions.
Eta expansion is a language feature that will automatically convert methods to functions where appropriate.
Partial Functions are a special type of Function with more control over what input they accept.
We can convert between partial functions and complete functions with the lift and unlift operations.

Good job for making it to the end of the post!

Happy Scala-ing!