Erlang uses recursion extensively – so lets do some classic Fibonacci sequence recursion code katas for Erlang. Note, that I define things a little differently than the Wikipedia example I just linked to. In the algorithms below, I have defined the first first two elements as = 1 and there is no N defined for 0. So: fib(1) = 1, fib(2)=1, and fib(N) where N > 2 = fib(N -1) + fib(N – 2).

For fun, lets try a few different algorithmic approaches to this sequence. The classic example of Fibonacci sequence recursion is a literal representation of the algorithm, which ends up being rather computationally intensive. It looks rather elegant but starts getting massively slow around N = 35. This is because N is recomputed recursively every time it is used!

fib_slow(1) -> 1;
fib_slow(2) -> 1;
fib_slow(N) -> fib_slow(N-1) + fib_slow(N-2).

Here’s an example of a different algorithm, where each value is accumulated in a list and looked up rather than recomputed. There also is a io:format print statement that returns the final sequence of N numbers. This method is a lot faster than the inefficient version above. You can set N = 1,000 without much fuss. If you comment out the print statement you can get to N = 20,000 speedily.

fib_list(1) -> io:format("~p~n",[[1]]);
fib_list(2) -> io:format("~p~n",[[1,1]]);
fib_list(N) when N > 2 -> 
    hd( fib_list1(N-1,[1,1]) ).

% delegate for numbers greater than 2
fib_list1(1,L) ->
    io:format("~p~n",[lists:reverse(L)]),
    L;
fib_list1(N,L) ->
    [H1,H2|_T] = L,
    fib_list1(N-1,[H1 + H2 | L]).

The final method computes the value but does not store it in a list. This one is the fastest (although I have not benchmarked them). This method also eliminates the risk of filling up memory with the list of previous numbers. N = 50,000 breezes by with this algorithm on my 2.2 Ghz Intel Dual Core MacBook Pro.

fib(1) -> 1;
fib(2) -> 1;
fib(N) when N > 2 -> fib1(N,1,1).

fib1(3,P1,P2) -> P1 + P2; 
fib1(N,P1,P2) ->
    fib1(N-1,P2, P1 + P2).

Here are all of the different approaches (in reverse orer) in a module.

-module(fib2).
-compile(export_all).
% get nth fibonacci no storage
% quite fast, get's nth number with no list storage

test() ->
    13 = fib(7),
    13 = fib_list(7),
    13 = fib_slow(7),
    horray.

fib(1) -> 1;
fib(2) -> 1;
fib(N) when N > 2 -> fib1(N,1,1).

fib1(3,P1,P2) -> P1 + P2; 
fib1(N,P1,P2) ->
    fib1(N-1,P2, P1 + P2).

% fib sequence made by accumulating values in a list
% pretty fast even at N = 20,000.
fib_list(1) -> io:format("~p~n",[[1]]);
fib_list(2) -> io:format("~p~n",[[1,1]]);
fib_list(N) when N > 2 -> 
    hd( fib_list1(N-1,[1,1]) ).

%delegate for numbers > 2
fib_list1(1,L) ->
    io:format("~p~n",[lists:reverse(L)]),
    L;
fib_list1(N,L) ->
    [H1,H2|_T] = L,
    fib_list1(N-1,[H1 + H2 | L]).

% ineficient recursive method for fibonacci computation
% very slow at N = 35
fib_slow(1) -> 1;
fib_slow(2) -> 1;
fib_slow(N) -> fib_slow(N-1) + fib_slow(N-2).

Erlang Pragmatic Studio with Joe Armstrong & Dave Thomas – Chicago, Feb 13-25, 2008.

Day 3 – Feb 14,2008.

Here are my notes from Day 3 of the Pragmatic Studio Erlang course. Note that these notes are only vaguely edited. Be aware that these are notes so there are certainly typos and errors.

Beginning of the Day

Always good to develop from the specific to the general. e.g. make a simple app, then parameterize it with functions. Universal server that evaluates F is a very powerful generalization of a specific application (e.g. messaging).

can send a message to your self:

self ! a. % Could tie into recursion

a lot of users Erlang guru’s (tailf, Kredita) don’t use OTP, as they participated in the OTP creation. But if you build up the Gen server from incremental steps, you may find it not so difficult. The question is, does OTP really fit the problem at hand.

If you do spawn_link from the shell, report errors back to the shell.

erl -bootstart sasle % spelling system application support library. Get error logger and a few other items.

sasle libraries were added into Erlang at a later date for historical reasons.

Bit Syntax

In ruby or other languages would be fairly tedious (ands, ors, bitshifts, etc). Erlang has a high level syntax that lets you specify the structure of binary data.

Red = 2, Green=61, Blue=20.
=> 20
Bin = <<Red:5,Green:6,Blue:5>>
=> <<23,180>>
io:format("~8.2.0B~8.2.0b~n", binary_to_list(Bin)).
=> 00010111 10110100

Bit Twiddling History:

Erlang had lots of protocols internally in Erickson. Decided to implement every RFC – but was too difficult. Lots of bit twiddling. Because of this they made a sub-language for doing this bit-twiddling so that this was implement any protocol. Optimal and efficient unpacking of datastructures.

1/3rd of details were in the Erlang book. The rest needs to be found in the man pages.

There are also bit comprehensions – generators of bit stream. Good for huffman encodings, MIPS processing, etc. Very efficient. This is in the experimental phase but will be integrated into the system. Undocumented items are not official and may be changed. Once the docs are written it’s official.

Erlang mailing list erlang.org > FAQs and mailing lists.

Book page 86 great shoutcast server for distributed audio. There is an excercise for bit processing, but we are skipping it in the interest of time.

file at a time I/O

Reading file at at time is about 10-20 times faster than reading line at a time. Erlang is not as efficient as some other languages for string matching. Use [H|T] to parse. Only multi-gig pattern are difficult for this. binary input and output is definitely fastest.

file:consult(File) -> {ok, Term} | {error, Why}

Term Access – read configuration file

-module(test4).
-
read_config() ->
    {ok,[{host,Host,port,Port}]} = file:consult(...)
    ...

Term IO

• inefficient

io:read('enter a term').
enter a term > {hello, "joe"}
io:get_line(...) %read from a shell

erl_scan:string("abc,123,{hello,joe}"). % turn into erlang tokens - very efficient
% written with explicit recursion that is more efficient than regex

file:pread ...

map onto standard file manipulations:

file:list_file(Dir)
file:delete()
...

filename:split(FileName) => [Component]

DNS servers are actually fairly easy. In a real DNS server there is tree walking for the sub-domain, domain, and tld xxx.xxxx.xxx

lots of DNS servers have interesting traffic patterns… tie ins with advertising sites, etc.

Erlang doesn’t link code into it’s Kernal. Need to establish a socket for C libs. Send messages using binary.

Orientation:

• nodes – when you say erl it starts a node. Doesn’t register this node by default.
• distributed erlang consists of a number of nodes that know about eachother.
• SMP takes advantage of multi-core – this is not distributed Erlang
• Socket distribution – is also not distributed Erlang

erl -sname %short name erl -name % different names

There is a magic cookie that they must share. Simple challenge request, traffic is not encrypted. Can communicate over SSH. DNS and name recognition systems are often poorly configured and this causes some difficulties for setting up distributed erlang apps. There are som FAQs on debugging distributed Erlang.

Three new primitives:

• spawn(Node, Mod, Func, Args) – links work the same
• alive(Node) – tell the system you are alive

Main libs:

• rcp
• global

Dave: mnesias cool be cause you can query it using list comprehensions.

Test on local nodes, then over a cluster.

erl -sname dave

rpc:call('joe@Daves-Powerbook', erlang, node, []).
rpc:call('joe@Daves-Powerbook', erlang, exit, [kill]).
rpc:call('joe@Daves-Powerbook', erlang, halt, []).

Shell commands for interacting with erl nodes:

• ^g gets you into the shell
• j – jobs
• r ‘dave@Daves-Powerbook’ % open a remote shell
• c 2 – change to the second process
• now at (dave@Daves-Powerbook)>
• toolbar:start(). % show current processes – queries OTP structure

Security:

• Security on distributed Erlang is very course grain. It’s all access or nothing.
• Perfect for a fire-walled corporate cluster.

Need to shore cookie to other machines. Check your home directory – .erlang.cookie

OTP – Open Telecomms Platform

This is the platform / framework for building Erlang applications.

• Error logs
• Hot swapping modules
• Common tasks that everyone needs
• etc.
• Mnesia
• SASL
• SNMP Agents
• Web Server

Lots of undocumented corners. Programming Erlang book is very helpful. It’s Open Source in the sense that you can see the source. But commit rights are controlled. It is product quality. There is no difference between Erickson’s products released and the Open Source distribution. There are 15 people working full time to maintain this – stable, battle tested.

OTP Principles

• Joe Armstron PHD Thesis
• erlang.org/doc/design_principles/part_frame.html
• client-server gen_server
• finite state machines gen_fsm
• event handling gen_event
• supervisor gen_sup
• Applications – e.g. mnesia, standard libraries, collections of processes shipped in it’s own right. start and stop etc.
• Releases – 4-5 applications combined form a product
• Application upgrade – upgrade a running system

Finite state machines

• State x Event -> State1 x Action
• Way of writing pattern matching on state and event.

gen_event

• no reply back (error log)
• event to new state, no reply

Order of reading:

• Joe’s thesis – best intro to OTP: sics.se/~joe/thesis
• erlang.org/doc/design_principles/part_frame.html
• forthcoming O’Reilly book

Behaviors:

• OTP name for “design patterns”
• Callback module

Case studies:

• AXD301 – 1.6 million lines of Erlang, built using OTP.
• Nortel Networks

Factoids:

• gen_server is the most used behavior.
• supervisor bridge – allows a C module to be fault tolerant in use with Erlang.
• gen_server – A generic client-server model

Types of Supervision:

• 1:1 vs 1:N supervision
• 1:1 – if child dies only 1 is restarted
• 1:N – if challed dies all child sibling processes restarted

trapexit.org is a good site.

Distributed applications:

• Must have the same beam code on all machines. There are checksums.
• Must have the same version of Erlang.

One way of solving this is to have the share the same backend NFS system.

Explicit code distribution. Send the code in a message:

{ok, Bin} = file:read_file("xxx.beam"),
Term = {apply, erlang, load_module, [Mod, Bin]}),
gen_tcp:send(Socket, term_to_binary(Term))

run it:

{apply, xxx, start, [...]}

dynamic code upgrade:

• Mod:Func(Args, ...) the latest version of Mod is called
• If you reload Mod then you can run two versions of the code at the same time
• You can only have two versions of the code running at the same time, an old and a new version
• To load a third version you must call erlang:purge_module(Mod) before reloading the module (think of this as a two place shift register)
• to load code call erlang:load_module(Mod, BeamCodeBin)

One process per whole Erlang instance on the OS. This would be very dangerous. But GC and as many things as possible should be written in Erlang. Hierarchy of tests:

• Unit test framework that comes from France (used by ejabber).
• Erickson has a test server that is distributed with Erlang. Massive regression and unit tests.
• Credita has modified to run test any time anything is checked into their repositories.
• Quick check written by John Hues. Was written in Haskell has been ported to Erling. Generates random tests that satisfy certain properties. Does some automatic reduction of code execution to minimize error conditions. This is an expensive use.
• E Unit – Dave found this somewhat cumbersom. Straight pattern matching was easier. Didn’t seem to have a huge value add. Would like to see how he can use differen’t naming conventions to automate and have some sort of runner. Find *.erl run anything named test and report if it doesn’t return a pass. Gen server seems easily testible by calling handler methods.

Not currently.

Petri Nets or activity diagrams?

Message sequence diagrams are used frequently. Joe is not a fan of drawing programs. Although, message sequence diagrams are extremely useful (Y = Time. X = Process). Joe is writing a program for animating Erlang.

erlang:trace(Pid, [Msgs, Call, list of things to trace]). 

This process get’s lots of messages. Joe is writing a program that stores these in a file and then animates the activities of these processes over time. Working with a games developer to animate this.

Erlyweb?

Joe – hasn’t spent much time with this and YAWS (Yet Another Web Server).

Erlang graphic user interface?

Lots written. Tend to be hard to use – wswidgets being used for 3D process modeling. Interfaces to SDL, GTK, Cairo, etc. TCL library is the mainly supported one that works everywhere. Lots of people have been doing GUIs in web browsers – action script 3 and flex. Can also use flex2 as a device driver for Erlang – e.g. video stream. Makes it possible to make these applications more portable. Therefore can use Flash as a device driver. Drop dead beautiful – flash with AIR may be the way to go.

Redeployable packaging?

Erlang is packaged for win, linux, and mac. CEAN (Comprehensive Erlang Application …) ... Wings. Martin Logan, faxian. Martin Logan also has generator applications for Erlang (Erlware?).

Good frameworks for doing ontological modeling and reasoning?

Expert systems shell. University of Corona. Reported at the Erlang user conferences. Multi agent listener but not generalized to owl.

List of reference projects?

No canonical reference. But check trapexit.org.

Where is Erlang center of gravity?

Attracting lots of interest in Financials – especially in London. Finance industry targeted conference coming up in London. Much bank interest – extreme real time demands. Number of pure erlang based trading companies. Debt buying company – purchases debt and collect. Buys these debts a few moments after the transaction. Mnesia founder involved.

Future changes – where is it going?

Not many changes to the language. Millions of legacy lines of code needed for compatibility – syntax unlikely to change. But, implementation speed optimization and multi-processor support is likely to improve.

Embedded Erlang?

4MB probably the minimum size. Used to be 640k but may be a better domain for C.

String localization – UTF8?

In a sense it is solved. In another sense it is not solved. In Erlang there is no string type – it is a list of integers. UTF8 is an integer and can be stored. However, the interpreter from text to binary and back is not written. However, it is not difficult. It is in fact a library change.

What is a good candidate for Erlang?

Telecomm applications, lots of small processes. Individual processes have low ammount of short computations. Good at coordination, concurrency, switching. Not good at matrix multiplication, gif encoding etc. Financial applications fit the erlang profile.

Libraries for interfacing to SQL databases?

ODBC libraries. Could just open a socket to the DB. Java has a good arbitrary support for sockets. Check Yariv’s Erly web implementation.

Summary

Fault tolerance drives a lot of the design decisions behind Erlang. Simple functional language. Function selection is by pattern matching Variables are immutable

Concurrent Erlang

• 3 primitives: Spawn, Send, Receive – very simplified from Object Oriented model.
• register / unregister can be used to associate a name with a process

Fault Tolerant

• catch .. throw, try … catch … end
• link, process_flag(trap_exit, true)

Distributed Erlang

• spawn(Node, Mod, Func, Args)
• or explicit term passing

Benefits of Erlang:

• Technologies becoming more concurrent and people are looking for solutions to that probelm. Cooperative applications, etc. Erlang is a working solution. Not operating system dependent. 20 – 30 years of experience with fault tolerance measured in years.
• Multi-Core ready
• Processes in the language

Some projects to research:

• AXD301 – biggest Erlang and Functional programming ever made 60 programmers x 3 years. Runs backbone of british telecom. Market leading in that sector. Peaked at 9×9s of reliability. This wasn’t a commercial project on 2-3% of the market.
• Kreditor (Kreditor.se) – buys debt. #2 IP startup in sweden. $150 year. Self funded. Founded by alumni of Blue Tail.
• SimpleDB (Amazon)
• CouchDB (text db)
• MociWeb (Mochimedia)
• Ejabberd (jabber server – by extension Twitter?)
• ErlyWeb (Erlang web framework inspired by Rails)

Erlang Pragmatic Studio with Joe Armstrong & Dave Thomas – Chicago, Feb 13-25, 2008.

Day 2 – Feb 14,2008.

Here are my notes from Day 2 of the Pragmatic Studio Erlang course. Note that these notes are only vaguely edited. Be aware that these are notes so there are certainly typos and errors.

Concurrency

review of an implementation of map:

map(F,[]) -> [];
map(F,[H|T]) ->
    [F(H) | map(F,T)].

Erlang is a concurrent language and a functional programming language second. Why concurrent? The world is concurent – external world & computing world. Often you inherit all the limits of the operating system in regards to processes. Erlang is not bound by the operating system. The emulator handles concurrency.

• Programming concurrent activities in sequential languages is artifically difficult.
• Shared memory (locks, mutexes, coroutines, processes, thread, deadlock, livelock, failure, thread-safe)
• Message passing

Web servers must spawn a new thread for every request. In Erlang when one thread crashes, only that one crashes not the whole system. This is not the case in other environments. Isolation of components is the key to the pattern.

Dave:

• process = object
• message = method
• can apply the same techniques in OOD. The small talk paradigm for passing messages unifies this (to a certain extent).

Joe: Alan Kay in his origination of object orientation, messaging was more primary. The problem with a function call is that the message must return to the caller.

Concurrency in other languages

• can only create very small number of processes
• heavy weight
• only supports message passing and not the kind of error handling semantics that Erlang has

If you know the name of a process, you can send it a message:

• Pid ! Message
• If Pid is hidden you cannot send a message to the process

Message arrives in a mailbox (like email, can be stacked). You never know if a message arrives (if you want to be sure send a message back).

Only three primitives for turning sequential program in to concurrent code:

• spawn – create new process
• send
• receive – try to match pattern

Spawn

Three syntaxes:

• spawn(fun foo/0) – runs foo/0 in a new process
• spawn(fun() -> ... end) – spawns and inline fun
• spawn(Mod, Func, [Arg1, Arg2, Argm]) performs

apply(Mod, Fun, [Arg1, Arg2, ..., Argn]) in a new process.

In Pid1 evaluate Pid2 = spawn(fun() -> ... end)

Send

• Pid ! Msg
• returns the message (e.g. A ! B ! C ! Msg)

Receive

same as a case statement:
receive ->
    pattern
    ...
end

self() is the pid of the current process. including it in a message allows the recipient to respond to us.

% receive whatever message comes first
receive
    Msg ->
    ...
end

% receive messages in order

receive
    foo -> true
end,
receive
    bar -> true % leave bar in the mailbox until foo is received
end

B!{transfer,self()}

receive
    {transfer, A} ->
        C ! {transfer, A}
    end

receive
...

promise and yield are non-primitive abstractions.

Are we hiding the wrong thing? Should we expose the protocol? Pros

• Hides message structure
• Makes module reusability easy
• we know how to define APIs

Cons

• More code
• We don’t know how to describe protocols
• Protocols do not exist in Erlang as first class objects

Key factor minimizing cross machine boundary cost.

Abstracting server functionality

• run a server that can run in the background
• needs some kind of state
• pass in state at the start
• in case of counter pass in 0
• then cast

Mutate state through recursion. Two more items.

Registered processes.

• Any program that wants to send a message to a Pid must have Pid in a local variable.
• Use register / unregister
• register(name, Pid), name ! Term sens a message to Pid
• whereis(Name) returns Pid or undefined
• mind the propigation delay

Example of register: register(counter, spawn(fun() -> counter(1)end)).

From shell see all the registered processor with: regs().

Registered names are global resources. If two processes register with the same name one will fail.

receive… after

wait for a certain period of time – timeouts

receive
    pattern1 -> ...
    pattern2 -> ...
after
    TimeInMilliseconds -> %can be atom {infinity}
        Actions
end.

Pid = spawn(fun() -> wait() end),
Pid ! {do, F}

wait() ->
    receive
        {do,F} ->
            F()
    end.

Security risks.

SMP and Processes

• Erlang can take advantage of multiple processes.
• Enable using -smp +S n options only if compiled with correct flags

all concurrent behaviour is programmed with the following (instead of blocks, mutexes, threads, etc…)

• spawn
• send
• receive

• default max processes is 327xx
• increase processes with: erl +P 1000000000
• once it passes

processes:max(1).

MACRO: Debugging
-define(DEBUGGING,true).
-ifdef(DEBUGGING).
-define(DEBUG(X),X).
-else
-define(DEBUG(X),void).
-endif

?DEBUG(io:format("Found:~p~n"),[Files]),
...

Two ways to do distributed computation:

• distributed Erlang as cluster with one name space. Consider different machines may have different versions of Erlang. Getting the right code onto the right machines may also cause issues. Fully connected machines don’t scale well – > 90 machines gets difficult.
• Explicit sockets. No issues of ownership. More easily scalable as other issues are encapsulated – memory is fully not shared.

Client

{ok, Socket} = gen_tcp:connect(Host, Port, [Options]),
    ok = gen_tcp:send(Socket,Data),
    ...
receive
    {tcp, Socket, Data} ->
        %% do something with the data
        ...
    {tcp_closed, Socket} ->
        %% take care of this ...

How does TCP work?

• server needs to be able to hand multiple clients trying to connect to it. The protocol is slightly different.
• c level api – listen(ip, port);

all that listen does is say I will support people on this given port. call accept and turn it into an IO object that you can use. Typically each connection spawns a different process.

a single-shot server accepts one connection: start_server()

{ok, Listen} = gen_tcp:listen(Port, ...)
spawn(fun() -> par_loop(Listen) end).

par_loop(Listen) ->
    {ok, Socket} = gen_tcp:accept(Listen),
    spawn(fun() -> par_loop(Listen) end),
    loop(Socket).

Packet lengths:

• A 4 byte length header is automatically added/removed by the system when calling gen_tcp:send and messages are assembled to the correct length before {tcp, Sock, Data} message are sent to the controlling process.

gen_tcp:connect(Host, Port, [..., {packet,4, ...}])
gen_tcp:listen ....

Sending Erlang terms
gen_tcp:send(Socket, term_to_binary(Term))...

receive
    {tcp, Socket, Data} ->
        Term = binary_to_term(Data),
        ...

The Middle Man Pattern: - Middle man has to be written for a particular protocol. - like unix filters (Dave) – piping

loop(Pid,Socket) ->
    receive
        {Pid,Msg} ->
            gen_tcp:send(Socket, term_to_binary(Msg)),
            loop(Pid, Socket);
        {tcp, Socket, Data} ->
            Pid ! binary_to_term(Data),
            loop(Pid, Socket);
        {'Exit',Pid} ->
            gen_tcp:close(Socket);
        {tcp_closed, Socket} ->
            exit(Pid, socket_closed)
        end.

use term_to_binary and binary_to_term.

when to use:

• , separate individual expressions (smallest scope)
• ; pattern matching
• . at end of function (biggest scope)

normally you don’t parallelize file read/write access – another case of shared memory.

% from simple_coordinate_server.erl
% pass in milliseconds or infinity to sleep forever
sleep(T) ->
    receive
    after T ->
        void 
    end.

gen_txp:listen(0 ...) % gets a free port

Concurrent patterns:

• client server model is the most used pattern when writing concurrent programs.
• can also use worker manager pattern.
• or finite state machine.

OTP course would require Erlang background.

try
...
catch
    exit:Why ->
        From ! {Name, die, why}, % die and tell calling process
        loop(State)
end

delegation is a nice place to do load balancing: client, delegator, responder

Erlang was developed in a trusted environment, many of the techniques are applicable to this environment.

This is an empty server that will run anything passed to it (guest account slightly safer but still incredibly permissive):

Pid = spawn(fun() -> wait() end),
...

Pid ! {do, F}
...
wait() ->
    receive
        {do, F} ->
            F()
        end

Many variations of concurrency:

• Resource managing
• Mobile Code
• Mobile code with transactions

Systems can be specified functionally.

• Systems have non-functional behavior (scalability, fault tolerance) – usually defined very weakly.
• These are usually vaguely defined.
• In Erlang these things can be specified clearly.

In Erlang you often write functional and non-functional code separately. These can then be used independently. Can abstract things like scalable fault-server and fault tolerant processing. Cool!

You can say how many failed processes are allowable in a given period. This can be specified programatically or in product requirements. e.g. if there are three machines and one machine fails and data is lost it is warranted, but if two machines die and data is lost it is out of warranty.

links

links define error propagation paths If A is linked to B then:

• If A fails then B will be notified
• If B fails then A will be notified

Signals and Messages:

• messages are sent with send
• signals are sent when processes die. Signals are not messages.
• if process A is linked to B it A will be sent an exit signal if B dies.

layers to make the system fault tolerant:

• application level
• level 2 – trap exits
• level 1 – core level to trap exits

considerations:

• if you do spawn followed by link the process might die before you get to the link statement
• spawn_link is like spawn followed by link only the two are performed atomically.
• trap_exit

principles of remote error handling

• fault tolerant systems need two separate computers
• let one process do the work
• let some other process fix the error, even remotely. If a whole machine is crashed it must be fixed remotely. This pattern should also be applied on a local system – cross machine boundary should be handled in the same manner for scalability.
• exit signals are like uncaught exceptions that escape from the process
• failing processes should fail early

How do you know when a machine fails? In distributed Erlang, there should be a heartbeat process that is sent between machines (1-10 times a second).

use halflink for monitoring.

Link methods:

• link
• unlink
• process_flag(trap_exit, true) – tap exits
• {‘EXIT’,Pid, Why} – message sent when a process dies These methods are orthogonal to spawn, send, receive