Aquabu - Home

Scruffy Ruby Gem Debugging

2008-07-16T09:25:00Z

!http://www.aquabu.com/assets/2008/7/16/cast.jpg! In the process of creating some graphs in ruby I decided to go upstream from ruport and use the scruffy gem directly. I made some discoveries along the way that touch on issues that could be encountered by various permutations of scruffy, macport, and rubygems users. h2. A Storm Brewing The first issue I encountered was this error:

testing_scruffy.rb:2:Warning: require_gem is obsolete.  Use gem instead.
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems.rb:304:in `report_activate_error': Could not find RubyGem Scruffy (>= 0.0.0) (Gem::LoadError)
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems.rb:238:in `activate'
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems.rb:76:in `active_gem_with_options'
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems.rb:61:in `require_gem'
	from testing_scruffy.rb:2

This seems like a straightforward case of not installing the gem. However, in my case the the gem *was* installed and there was additional strange behavior: require 'scruffy' would load the library in irb but not from a script (even with require 'rubygems' before it).This seemed inconsistent because $LOAD_PATH (aka $:) variables were identical in irb and in the script. After investigating a number of possibilities I updated rubygems (which was at 0.9.4) to 1.2.0 and this seem to resolve the blatantly bizarre behavior. You can check which version of ruby gems you are running with "gem -v". If you are using macports you may have an older version of ruby gems, in which case you may find it useful to update rb-rubygems. Caveat emptor: dependent updates may also be installed - such as ruby. Here's how you can update your macport rb-rubygems library on OS X:

sudo port upgrade rb-rubygems

h2. Scruffy Needs A Shave If you do upgrade ruby gems you may run into the another issue. With an updated version of ruby gems the require_gem method is obsolete. Because of this you may see the following error since scruffy 0.2.2 is not updated to use the *gem* method instead of *require_gem*:

NoMethodError: undefined method `require_gem' for main:Object
	from /opt/local/lib/ruby/gems/1.8/gems/scruffy-0.2.2/lib/scruffy.rb:18
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:32:in `gem_original_require'
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:32:in `require'
	from (irb):3

I used the hack below to get around this and to avoid unpacking the gem in to my project directory for modifications. Once again caveat emptor - you may ultimately be better off unpacking (freezing) the gem into a local directory and including it in your application directly. Here's the hack which simply creates an alias to suspend the deprecation of the require_gem method indefinitely:

require 'rubygems'
alias require_gem gem #later versions of gem use gem instead of require_gem
require 'scruffy'

h2. Case Of The Missing Pie Ah, but that's not all. It appears that tutorials and documentation are out of sync with the rubygem download... see example code at "http://scruffy.rubyforge.org/":http://scruffy.rubyforge.org/ which suggests this usage (with my require hack at the top):

require 'rubygems'
alias require_gem gem
require 'scruffy'

graph = Scruffy::Graph.new
graph.title = "Favourite Snacks"
graph.renderer = Scruffy::Renderers::Pie.new

graph.add :pie, '', {
      'Apple' => 20,
      'Banana' => 100,
      'Orange' => 70,
      'Taco' => 30
}

graph.render :to => "pie_test.svg"
graph.render :width => 300, :height => 200, :to => "pie_test.png", :as => 'png'

Running the sample code throws this error:

test_scruffy.rb:7: uninitialized constant Scruffy::Renderers::Pie (NameError)

Well, this error occurs because there is no such class as Scruffy::Renderers::Pie. It's not in the docs and if you do grep -r "pie" . from within the scruffy-0.2.2 folder it comes up with nothing that would lead to the creation of such a class. FYI, the docs are visible online here "http://scruffy.rubyforge.org/doc/":http://scruffy.rubyforge.org/doc/ Well, it's hard to use a class that doesn't exist. It appears that this may be present in scruffy 0.2.3 released on July 13th - but as of this writing it's not downloaded via sudo gem update scruffy. If you need this and it it is not yet available through ruby gems - you may need to do a manual install. In the mean time let's try something else instead. h2. Steady Into Port (Mind The Wireshark)

graph = Scruffy::Graph.new
graph.title = "Sample Line Graph"
graph.renderer = Scruffy::Renderers::Standard.new

graph.add :line, 'Example', [20, 100, 70, 30, 106]

graph.render :to => "line_test.svg"
graph.render  :width => 300, :height => 200, :to => "line_test.png", :as => 'png'

Well if you have RMagick installed that may have worked for you. If not you will need to make sure rmagick is installed. On my machine I had to remove some dependencies before reinstalling tiff with different compilation options. Here are the commands I ran. You will probably have different dependencies, so know what you are doing. Don't just blindly run these commands or you may uninstall something you need:

sudo port uninstall wireshark
sudo port uninstall gtk2
sudo port uninstall tiff @3.8.2_1+macosx
sudo port install tiff -macosx imagemagick +q8 +gs +wmf
sudo gem install rmagick

Note that installing and compiling rmagick took several minutes. I would suggest looking at the "rmagick installation page":http://rmagick.rubyforge.org/install-osx.html if you need to install this. At this point you should be able to output your svg and png files. h2. Conclusion Careful, or due to "environmental conditions" you could set out expecting a simple three hour tour happily outputting beautifully rendered svgs as you sail effortlessly through graph land; but instead, end up whistling along sympathetically to the Gilligan's Island theme song as you set ground on the shore of an uncharted desert isle. Hopefully this will help you through some of the things you may run into, and you can have a smooth sail enjoying scruffy.

Proc.new, lambda, and proc

2008-06-18T17:42:00Z

In response to the "harmony of thought posting on good ruby interview questions":http://blog.ritirisi.com/2008/06/17/15-questions-to-ask-during-a-ruby-interview I wrote the following (posted here for interest and to preserve formatting)... One thing about lambdas and procs. Even though they appear to be the same class they do not behave completely the same. From running my_lambda.inspect and seeing the Proc instance you would never guess it but they are different. Essentially lambda and Proc.new handle arguments and the "return" statement differently. Also the "proc" method returns a lambda not a Proc. Here's some code to prove the point. The argument handling is the easiest to demo:

my_proc = Proc.new {|x,y| print x,y}
my_lambda = lambda {|x,y| print x,y}
my_other_proc = proc {|x,y| print x,y}

# pass in one variable instead of 2
my_proc.call(1) #this doesn't throw an error and sets y to nil
my_lambda.call(1) #this throws an error
my_other_proc.call(1) #this throws an error like a lambda because it actually is a lambda not a proc in ruby 1.8.6

Also in Ruby 1.9 I think that the "proc" method (as seen in line three) will actually return a Proc and not a lambda. Ruby 1.9 will also add a "lambda?" method for you to clarify the difference between a Proc and a lambda. O'Reilly's book "The Ruby Programming Language" by Matz (Yukihiro Matsumoto) and David Flanagen has a great chapter on this.

Railsconf 2008 Scribblings: Hosting and the Woes (Engine Yard Panel)

2008-06-13T23:43:00Z

5/30/2008, 11:45AM, Portland, OR
Here are some of my notes from Railsconf 2008. Warning these notes are only vaguely edited (but arguably still useful). There are certainly typos and errors. If executing code from one of these articles, your mileage may vary (from spontaneous self destruction of everything you know and love to spontaneous coolness). Also, you may want to double check “facts”. You definitely should not use these articles as a spelling reference. Feel free to post corrections. Engine Yard Panel: * Tom Mornini - CTO * Taylor Weibley - Application Support Director * Edward Muller - Automation Manager (fluent in binary) * Ezra Zygomowitz - System Architect * Jamie van Dyke Problems they have had and how they solved most of them. Then questions and answers. h2. Main problem with rails is active_record Great for quickly building with RESTful resources. But... * find(:all).each do |leak| * testing is usually with 20 records. * no indexes. Engineyard has a team of SQL guys to optimize in this area. One of the first problem they noticed that you need indexes on all foreign keys. If you don't have them they are full table scans. Migrations automatically index ids but not foreign keys. Basically anything that ends with _id needs an index. There may be some database experts that will disagree but this is not usually the case. Use explain in mysql /G to make sure that the indexes are being used in the queries. Take a look at the new Relic optimization stuff it's super cool. h2. Plugins that give common problems * Most common and hated plugin is ferret. Indexes grow past 1G and indexes frequently corrupted. Use sphinx instead. Sphinx will just update indexes on the delta. * Image Science uses ruby inline which needs environment variables to be set. Can be used perfectly in development and then when they shift to production monit clears those variables out. Not a huge problem but you should be aware of it. * Hodel 3000 loggers default is a verbose logging tool and it can generate huge log files. Adjust settings. Logging should be set to info. h2. How much traffic can you take? *Depends on coding* General stats: * Digg 10,0000+ visitors and 1% signup. What should you do? Cache it. At engine yard a slice could handle this quite easily. Customers are often concerned with this sudden out of the blue Digg or TechCrunch. But in reality you usually are and there is no surprise. Makes it so you can be ahead of the curve. * TechCrunch 1000+ visitors but they are more interested in what you are doing. * Ruby Inside / Flow 500 - 1000 visitors a day, also don't get many sign ups. It's more about what can the visitor do. * The Today Show - 100,000+ visitors in the first hour or so. Can get up to 10,000 signups in the first couple hours. You will have advance warning which is great. * Fox Business - 2000+ connections. h2. How many queries can your site take? It depends on code. * Queries per page * But you should cache! * Best way to handle hike in traffic is cache / memcache * Keep file input output down as low as possible * Separate out to different disks, you don't want simultaneous reads and writes h2. Deployment Lots of customers not up to speed on Capistrano, git / subversion. Engine yard gives deploy.rb (shameless plug from them :-) gem source - a http://www.engineyard.com ... Includes cap examples list of extra tasks - loads of things that it can do. h2. ebb | mongrel | thin. Which do customers prefer? Alternatives to mongrel are possible but not necessarily production ready (ebb). Don't need to be super concerned about it unless you are doing 1,000 hits per second. The bottleneck is usually the user code, this is far more relevent. That said, thin may be 13% faster but you need to be in the top 5% of internet traffic for this to matter. h2. Audience Q&A What happened with NGINX? You haven't heard about much new because it's in use and it just works. They haven't really had any problems with NGINX. Some minor issues but they have been ironed very quickly. They have seen 40MB/sec of static images served and not show up top command. Interpreters?
Rubinius and Modrubinius are going to be the way to go in the future. Thin instead of mongrel because of lower memory footprint?
There are a lot reasons to save memory but mongrel is a minor concern. More than 3-4 mongrels per CPU is pointless. The bottleneck is file io rather than memory. A common misconception is that you need more mongrels. Thin tends to backup processes more than mongrel. It is important to watch the real behavior of the app because every app uses resources differently. Should static files be local or shared across cluster?
Cluster is fine @ Engine Yard, but depends on file system. NFS was cool in 1979... How to handle asynchronous background processes?
Background Job (BJ) is pleasurable to work with. Written by NGINX. Do you leave keep alive with NGINX?
Untested. But historical use with apache pre-rails was not succesful. Seperate server for search workload? How should resources be distributed across server farm and how does this map to virtualization? Engine Yard are strong believers of virtualization. Intended from earlier on to only have virtualized servers. The more that they have gotten involved in it they have discovered that if you have a well built virtualization environment you quickly get the sense that you are operating on separate machines. NGINX, 3 Mongrels, and an memcache per instance. If you have a solid architecture you could have databases on multiple instances. Single server rack at engine yard with 18 servers would run bottom 97% of all internet apps. There's huge CPU capacity in a virtualization environment. CPUs are not typically the bottleneck (unless you are doing fluid dynamic calculations). What is the Engine Yard uptime?
3-4 9s up time. Rack Space rates using a questionable rating system.

Railsconf 2008 Scribblings - MagLev: Ruby That Scales

2008-06-13T23:26:00Z

Talk with Avi Bryant and Bob Walker
5/30/2008, 3:40PM, Portland, OR
Here are some of my notes from Railsconf 2008. Warning these notes are only vaguely edited (but arguably still useful). There are certainly typos and errors. If executing code from one of these articles, your mileage may vary (from spontaneous self destruction of everything you know and love to spontaneous coolness). Also, you may want to double check "facts". You definitely should not use these articles as a spelling reference. Feel free to post corrections. MagLev is rails that go fast. Typically have mongrel cluster talking to storage device and memcache instance for each server. Maglev unifies this stack: Storage, Cache, and Ruby Example: "Ordinary ruby shells". Two terminals using the same shared objects! You can do this for any number of boxes across the entire system. Local state is not shared, global variables ($hat). Ruby shell has transactions as you are used them. Transactions are there for mutable state.Commits don't only go to in memory caching, they go to disk and if there is a failure then the machines can be rebooted with the existing state. Could this be on multiple machines? Yes. The storage can be greater than ram because they can be persisted to disk. The system does not currently run rails, but it will be soon. Can currently run webrick with hello world. Question: Is the ruby spec implemented and at what level? Next phase is implementing the ruby specs against this. Question: What happens to files and sockets? Each VM will have it's own seperate copy of that resource and they will be nil'ed out when brought over. Question: Is this a written from scratch ruby machine? Yes, it is not MRI. Gemstone founded in 1982 in Portland. * Cretated GemStone smalltalk * Enterprise proven technology * Production systems deployed worldwide * Support domains that are not well served by relational database systems * Worldwide container * Intelligence * Telecom * Etc. MagLev * 1 dual core up to really large systems * Mutliple ruby virtual machines * Oject Caches/Repository * Transactional * Distributed * Scalable They are currently 100 days into the implementation process. * Dynamic Language VM * Based on Gemsone/s 64VM * Well tuned garbage collection * Extended with new byte codes for Ruby * JIT native code generation Object Caching * Shared caching uses shared memory * Repository objects are read into the cache ... Transactional * Fully recoverable ACID Transactions * Simple API * Conflict management Persistence * Transparent * Native Language * Simple API * You can store a trillion instances * Up to 17PB Benchmarks * bm_so_exceptions 5.6 x faster than MRI * fib 12.5 x MRI * vm2 method 24 x MRI * bm_so_sieve 68.4 times * bm_loop_whileloop2 111.5 x MRI "Holy Shit" -Antonio Gangiano (Great Ruby Shootout) Numbers will be independently verified shortly. Audience question. What is missing from Ruby that will decrease this performance? The plan is ruby spec and core library. But does it scale? Yes. What's the vision? Rails - taking it to a different level of scalability. What Else? * Open Community Involvement * Ruby Spec Compliance * MagLev Ruby open source * Free version of MagLev will be feature limited for small to medium scale * Info @ maglev.gemstone.come A lot of rails makes use of the relational database structure and indices. Will Maglev address these? Yes. And you will be able to make your queries in the native language. Will likely clone the API but run gemstone underneath. Continuations? Yes. They are supported in smalltalk. How to optomize? Will it be a linear scan? Gemstone does will do an linear scan for a basic select but gemstone does have various indexing. Will the Gemstone VM be open sourced (asked by Charles Nutter from jRuby team): Bob: "In a world of infinite possibilities it's possible." Will existing extensions in C be possible to integrate? Not currently. Rubinius has done some good work in this area and it may be possible to use a similar approach for Gemstone. C extensions may not be necisary in certain cases. However rails depends on C like BigDecimal(Wilson)? Will solve those problems when they get to them. ActiveRecord uses SQL fragments. How would that work since it's not part of ruby? To port a rails app may require getting rid of optomizations done in sql.... this is an area where feedback will be welcome. This is yet to be designed built. Which pieces are open source? All of the stuff at the ruby level is hopefully to be shared with Rubinius. ActiveRecord API will be. But as for the underlying VM Bob answered that earlier (e.g. "in a world of infinite possiblities...."). Is the persistent store distributable? More like typical SQL statement. Master slave replication... How is the VM implemented? In short... in C. When is going to be ready? Pay attention to what we are doing. More info in September.

Railsconf 2008 Scribblings: Joel Spolsky (Keynote)

2008-06-13T23:17:00Z

5/30/2008, 9:15am, Portland, OR Here are some of my notes from Railsconf 2008. Warning these notes are only vaguely edited (but arguably still useful). There are certainly typos and errors. If executing code from one of these articles, your mileage may vary (from spontaneous self destruction of everything you know and love to spontaneous coolness). Also, you may want to double check "facts". You definitely should not use these articles as a spelling reference. Feel free to post corrections. Intro by DHH on Joel: * Joel is author of Joel On Software * Generated programming language to generate other programming language * fighting complexity (anti-architect astronaut) h2. Joel's Talk Begins... Why does Brad Pitt make so much more money and have a bigger brand than Ian Somerhalder? What becomes "blue chip software" rather than off brand? Herman Miller Aeron Chair was the bluechip chair. All types of people make clones. Clones didn't have market share. This happens in every field. Happens in music as well. Great software has three components that make them blue chip: * Make people happy * Obsess over aesthetics * Observing the culture code h2. Make People Happy (Agency) [Very funny example about logging in to windows to post a photo and then having to install windows downloads and then use original install a file from the insert original install disk. In a word you feel enslaved (you are not in control of the process)] Martin Seligman pioneering psychologist of "Learned Helplessness" liked to study happy people which is rare amongst psychologists. Example: Someone dies > you are stuck in bed depressed > you loose your job and that you begin to believe that you lack agency and then you begin to teach yourself that you are helpless. Seligman's strategy was to teach people that they have power over there environment through tricking the brain into a virtuous cycle of having agency (initially through accomplishment of simple tasks). Abercrombie ecommerce checkout forces you through 4 pages. Amazon gives you all the options at once (change address, shipping options, etc). This is Joel's first example of giving the user agency. Make people happy by putting them in control (speed of response is also important here - the business case for AJAX). h2. Obsess over aesthetics Samsumg Blackjack vs iPhone. * Blackjack faster * iphone bricks if you install third party software * Blackjack ugly & iphone is seamless and shiny (if you accidentally swallowed one it would go right down) * Blackjack replaceable battery - iphone doesn't (avoids the non aesthetic seam). Another example is looking at the bottom of the MacBook vs. thinkpad. Apple design decisions are about fashion. Another example is a historical Paris apartment building in Paris has no fire escapes. It turns out fire escapes are zoned, but the appartment buildings are reclasified as monuments and they are not inspected. This is another example of taking aesthetics as more important than core functionality. Programers call this lipstick and say I want to work in the guts (the 90% of the iceberg below the surface). But it turns out that aesthetics are very important. To programmers it is not clear why artists such as Basquaiat etc. sell paintings for $50+ million. Specifically programmers don't get modernist architecture. Modernist principles remove all decoration from architecture. There is a parallel here between the programers gravitation to the command line vs. glossy operating systems. h2. The Culture Code Example: the promotion of sport utility vehicles. Deaths per million cars: * Toyota Camry 41 * Ford Explorer 88 This has to do with a number of factors. But you feel safe in these vehicals. This has to do with promotion (see the culture code). * Everything around you should be round and soft * You should be up high (on the reptilion level if you feel tall you feel you are dominating) * Key element of safety as a child is being fed if everything is round, soft, high, and you feel fed (cupholders) you feel safe. Another example like this is the word "Enterprise". Recalls tokyo at night, a clean server room, the starship enterprise, people in clean business suits looking up(image). Another example is Web 2.0. Lots of logos. They don't have visions they go to parties... etc. Why the luck stiff :-) How could beauty, happiness, motivation, pride, pleaseure, enthusiasm be involved in coding. Very funny counter examples for python and java go here... This all boils down to Misattribution. You have a physical reaction and you think that it has something to do with the mental plane but it doesn't. People on the right hand side of the movie have 10% greater ratings than the left.

Installing vim with +ruby or --enable-rubyinterp on OS X 10.5.2

2008-05-27T07:55:00Z

The problem: When installing vim from source with --enable-rubyinterp option you are receiving numerous unsolvable errors (such as ncurses cannot be found when it is installed and you are directing ./configure to it using --with-tlib ncurses). Or, you are trying to install vim +ruby using MacPorts and you are getting similar errors. The solution: install vim using MacPorts and install Ruby first.

sudo port install ruby
sudo port install vim +ruby +huge

Newly installed version is in /opt/local/var/macports/software/vim/7.1.270_0+huge+ruby/opt/local/bin After launching this version of vim it responds to:

:ruby puts "hello world!"

If you have the following in your .bash_profile or .profile file and you reopen your terminal, the newly installed MacPort will take precedence over the default OS X install of vim:

export PATH=/opt/local/bin:/opt/local/sbin:$PATH

Note that having your MacPort ruby install take precedence over your default ruby installation can modify your environment. Particularly, it may cause ruby to look for a different gem library. You can verify you are using the correct configuration by checking your gem path, version, and gems that you have installed:

which gem
gem -v
gem list --local

If your MacPorts gem version is taking precedence over other gem libraries that you would prefer to use, you can modify the file name or remove the version of the gem that you do not want to use. It is likely that MacPorts is referencing /opt/local/bin/gem and you would like it to reference /usr/bin/gem.

Fibonacci Sequence Recursion in Erlang

2008-02-16T20:00:00Z

Erlang uses recursion extensively - so lets do some classic "Fibonacci sequence":http://en.wikipedia.org/wiki/Fibonacci_number recursion code katas for Erlang. Note, that I define things a little differently than the Wikipedia example I just linked to. In the algorithms below, I have defined the first first two elements as = 1 and there is no N defined for 0. So: fib(1) = 1, fib(2)=1, and fib(N) where N > 2 = fib(N -1) + fib(N - 2). For fun, lets try a few different algorithmic approaches to this sequence. The classic example of Fibonacci sequence recursion is a literal representation of the algorithm, which ends up being rather computationally intensive. It looks rather elegant but starts getting massively slow around N = 35. This is because N is recomputed recursively every time it is used!

fib_slow(1) -> 1;
fib_slow(2) -> 1;
fib_slow(N) -> fib_slow(N-1) + fib_slow(N-2).

Here's an example of a different algorithm, where each value is accumulated in a list and looked up rather than recomputed. There also is a io:format print statement that returns the final sequence of N numbers. This method is a lot faster than the inefficient version above. You can set N = 1,000 without much fuss. If you comment out the print statement you can get to N = 20,000 speedily.

fib_list(1) -> io:format("~p~n",[[1]]);
fib_list(2) -> io:format("~p~n",[[1,1]]);
fib_list(N) when N > 2 -> 
	hd( fib_list1(N-1,[1,1]) ).

% delegate for numbers greater than 2
fib_list1(1,L) ->
	io:format("~p~n",[lists:reverse(L)]),
	L;
fib_list1(N,L) ->
	[H1,H2|_T] = L,
	fib_list1(N-1,[H1 + H2 | L]).

The final method computes the value but does not store it in a list. This one is the fastest (although I have not benchmarked them). This method also eliminates the risk of filling up memory with the list of previous numbers. N = 50,000 breezes by with this algorithm on my 2.2 Ghz Intel Dual Core MacBook Pro.

fib(1) -> 1;
fib(2) -> 1;
fib(N) when N > 2 -> fib1(N,1,1).

fib1(3,P1,P2) -> P1 + P2; 
fib1(N,P1,P2) ->
	fib1(N-1,P2, P1 + P2).

Here are all of the different approaches (in reverse orer) in a module.

-module(fib2).
-compile(export_all).
% get nth fibonacci no storage
% quite fast, get's nth number with no list storage

test() ->
	13 = fib(7),
	13 = fib_list(7),
	13 = fib_slow(7),
	horray.

fib(1) -> 1;
fib(2) -> 1;
fib(N) when N > 2 -> fib1(N,1,1).

fib1(3,P1,P2) -> P1 + P2; 
fib1(N,P1,P2) ->
	fib1(N-1,P2, P1 + P2).
	
% fib sequence made by accumulating values in a list
% pretty fast even at N = 20,000.
fib_list(1) -> io:format("~p~n",[[1]]);
fib_list(2) -> io:format("~p~n",[[1,1]]);
fib_list(N) when N > 2 -> 
	hd( fib_list1(N-1,[1,1]) ).

%delegate for numbers > 2
fib_list1(1,L) ->
	io:format("~p~n",[lists:reverse(L)]),
	L;
fib_list1(N,L) ->
	[H1,H2|_T] = L,
	fib_list1(N-1,[H1 + H2 | L]).

% ineficient recursive method for fibonacci computation
% very slow at N = 35
fib_slow(1) -> 1;
fib_slow(2) -> 1;
fib_slow(N) -> fib_slow(N-1) + fib_slow(N-2).

Erlang Pragmatic Studio - Day 3 Notes

2008-02-15T19:07:00Z

Erlang Pragmatic Studio with Joe Armstrong & Dave Thomas – Chicago, Feb 13-25, 2008.

Day 3 – Feb 14,2008.

Here are my notes from Day 3 of the Pragmatic Studio Erlang course. Note that these notes are only vaguely edited. Be aware that these are notes so there are certainly typos and errors.

Beginning of the Day

Always good to develop from the specific to the general. e.g. make a simple app, then parameterize it with functions. Universal server that evaluates F is a very powerful generalization of a specific application (e.g. messaging).

can send a message to your self:

self ! a. % Could tie into recursion

a lot of users Erlang guru’s (tailf, Kredita) don’t use OTP, as they participated in the OTP creation. But if you build up the Gen server from incremental steps, you may find it not so difficult. The question is, does OTP really fit the problem at hand.

If you do spawn_link from the shell, report errors back to the shell.

erl -bootstart sasle % spelling system application support library. Get error logger and a few other items.

sasle libraries were added into Erlang at a later date for historical reasons.

Bit Syntax

In ruby or other languages would be fairly tedious (ands, ors, bitshifts, etc). Erlang has a high level syntax that lets you specify the structure of binary data.

Red = 2, Green=61, Blue=20.
=> 20
Bin = <<red:5>>
=> <<23>>
io:format("~8.2.0B~8.2.0b~n", binary_to_list(Bin)).
=> 00010111 10110100

Bit Twiddling History:

Erlang had lots of protocols internally in Erickson. Decided to implement every RFC – but was too difficult. Lots of bit twiddling. Because of this they made a sub-language for doing this bit-twiddling so that this was implement any protocol. Optimal and efficient unpacking of datastructures.

1/3rd of details were in the Erlang book. The rest needs to be found in the man pages.

There are also bit comprehensions – generators of bit stream. Good for huffman encodings, MIPS processing, etc. Very efficient. This is in the experimental phase but will be integrated into the system. Undocumented items are not official and may be changed. Once the docs are written it’s official.

Erlang mailing list erlang.org > FAQs and mailing lists.

Book page 86 great shoutcast server for distributed audio. There is an excercise for bit processing, but we are skipping it in the interest of time.

file at a time I/O

Reading file at at time is about 10-20 times faster than reading line at a time. Erlang is not as efficient as some other languages for string matching. Use [H|T] to parse. Only multi-gig pattern are difficult for this. binary input and output is definitely fastest.

file:consult(File) -> {ok, Term} | {error, Why}

Term Access – read configuration file

-module(test4).
-
read_config() ->
    {ok,[{host,Host,port,Port}]} = file:consult(...)
    ...

Term IO

inefficient

io:read('enter a term').
enter a term > {hello, "joe"}
io:get_line(...) %read from a shell

erl_scan:string("abc,123,{hello,joe}"). % turn into erlang tokens - very efficient
% written with explicit recursion that is more efficient than regex

file:pread ...

map onto standard file manipulations:

file:list_file(Dir)
file:delete()
...

filename:split(FileName) => [Component]

DNS servers are actually fairly easy. In a real DNS server there is tree walking for the sub-domain, domain, and tld xxx.xxxx.xxx

lots of DNS servers have interesting traffic patterns… tie ins with advertising sites, etc.

Erlang doesn’t link code into it’s Kernal. Need to establish a socket for C libs. Send messages using binary.

Orientation:

nodes – when you say erl it starts a node. Doesn’t register this node by default.
distributed erlang consists of a number of nodes that know about eachother.
SMP takes advantage of multi-core – this is not distributed Erlang
Socket distribution – is also not distributed Erlang

same network: erl -sname %short name erl -name % different names

There is a magic cookie that they must share. Simple challenge request, traffic is not encrypted. Can communicate over SSH. DNS and name recognition systems are often poorly configured and this causes some difficulties for setting up distributed erlang apps. There are som FAQs on debugging distributed Erlang.

Three new primitives:

spawn(Node, Mod, Func, Args) – links work the same
alive(Node) – tell the system you are alive

Main libs:

rcp
global

Test on local nodes, then over a cluster. Dave: mnesias cool be cause you can query it using list comprehensions.

erl -sname dave

rpc:call('joe@Daves-Powerbook', erlang, node, []).
rpc:call('joe@Daves-Powerbook', erlang, exit, [kill]).
rpc:call('joe@Daves-Powerbook', erlang, halt, []).

Shell commands for interacting with erl nodes:

^g gets you into the shell
j – jobs
r ‘dave@Daves-Powerbook’ % open a remote shell
c 2 – change to the second process
now at (dave@Daves-Powerbook)>
toolbar:start(). % show current processes – queries OTP structure

Security:

Security on distributed Erlang is very course grain. It’s all access or nothing.
Perfect for a fire-walled corporate cluster.

Need to shore cookie to other machines. Check your home directory – .erlang.cookie

OTP – Open Telecomms Platform

This is the platform / framework for building Erlang applications.

Error logs
Hot swapping modules
Common tasks that everyone needs
etc.
Mnesia
SASL
SNMP Agents
Web Server

Lots of undocumented corners. Programming Erlang book is very helpful. It’s Open Source in the sense that you can see the source. But commit rights are controlled. It is product quality. There is no difference between Erickson’s products released and the Open Source distribution. There are 15 people working full time to maintain this – stable, battle tested.

OTP Principles

Joe Armstron PHD Thesis
erlang.org/doc/design_principles/part_frame.html
client-server gen_server
finite state machines gen_fsm
event handling gen_event
supervisor gen_sup
Applications – e.g. mnesia, standard libraries, collections of processes shipped in it’s own right. start and stop etc.
Releases – 4-5 applications combined form a product
Application upgrade – upgrade a running system

Finite state machines

State x Event -> State1 x Action
Way of writing pattern matching on state and event.

gen_event

no reply back (error log)
event to new state, no reply

Order of reading:

Joe’s thesis – best intro to OTP: sics.se/~joe/thesis
erlang.org/doc/design_principles/part_frame.html
forthcoming O’Reilly book

Behaviors:

OTP name for “design patterns”
Callback module

Case studies:

AXD301 – 1.6 million lines of Erlang, built using OTP.
Nortel Networks

Factoids:

gen_server is the most used behavior.
supervisor bridge – allows a C module to be fault tolerant in use with Erlang.
gen_server – A generic client-server model

Types of Supervision:

1:1 vs 1:N supervision
1:1 – if child dies only 1 is restarted
1:N – if challed dies all child sibling processes restarted

trapexit.org is a good site.

Distributed applications:

Must have the same beam code on all machines. There are checksums.
Must have the same version of Erlang.

One way of solving this is to have the share the same backend NFS system.

Explicit code distribution. Send the code in a message:

{ok, Bin} = file:read_file("xxx.beam"),
Term = {apply, erlang, load_module, [Mod, Bin]}),
gen_tcp:send(Socket, term_to_binary(Term))

run it:

{apply, xxx, start, [...]}

dynamic code upgrade:

Mod:Func(Args, ...) the latest version of Mod is called
If you reload Mod then you can run two versions of the code at the same time
You can only have two versions of the code running at the same time, an old and a new version
To load a third version you must call erlang:purge_module(Mod) before reloading the module (think of this as a two place shift register)
to load code call erlang:load_module(Mod, BeamCodeBin)

You can find all the number of processes running a particular version of a module. Some may never change, those you may have to kill.

When you load new code you get old and new code running at the same time. You must call purge_code before you add a third version of the code. This is going to effect 0.0x of people using erlang. After a crash you need to look at these issues.

This code swapping is not well documented. Mailing list may be helpful for this area.

Q & A With Joe Armstrong

What is the relationship between processes on the OS and Erlang? One process per whole Erlang instance on the OS.

When you spawn there is a run queue. The process that currently gets to the top of the queue. Every time you do a function call it’s called a reduction. If it’s waiting to receive a message, it’s taken out of the scheduling queue and is put into the waiting for message queue. If mail is delivered it’s put back in the run queue. There is no polling that is used. When you send a message to a processor and it hasn’t used all it’s reductions – may preempt …

Consider exposing schedule to Erlang? This would be very dangerous. But GC and as many things as possible should be written in Erlang.

What do you do in an app to seperate tests into a seperate file? Hierarchy of tests:

Unit test framework that comes from France (used by ejabber).

Erickson has a test server that is distributed with Erlang. Massive regression and unit tests.

Credita has modified to run test any time anything is checked into their repositories.

Quick check written by John Hues. Was written in Haskell has been ported to Erling. Generates random tests that satisfy certain properties. Does some automatic reduction of code execution to minimize error conditions. This is an expensive use.

E Unit – Dave found this somewhat cumbersom. Straight pattern matching was easier. Didn’t seem to have a huge value add. Would like to see how he can use differen’t naming conventions to automate and have some sort of runner. Find *.erl run anything named test and report if it doesn’t return a pass. Gen server seems easily testible by calling handler methods.

Diagraming conventions for Erlang? Not currently.

Petri Nets or activity diagrams?

Message sequence diagrams are used frequently. Joe is not a fan of drawing programs. Although, message sequence diagrams are extremely useful (Y = Time. X = Process). Joe is writing a program for animating Erlang.

erlang:trace(Pid, [Msgs, Call, list of things to trace]).

This process get’s lots of messages. Joe is writing a program that stores these in a file and then animates the activities of these processes over time. Working with a games developer to animate this.

Erlyweb?

Joe – hasn’t spent much time with this and YAWS (Yet Another Web Server).

Erlang graphic user interface?

Lots written. Tend to be hard to use – wswidgets being used for 3D process modeling. Interfaces to SDL, GTK, Cairo, etc. TCL library is the mainly supported one that works everywhere. Lots of people have been doing GUIs in web browsers – action script 3 and flex. Can also use flex2 as a device driver for Erlang – e.g. video stream. Makes it possible to make these applications more portable. Therefore can use Flash as a device driver. Drop dead beautiful – flash with AIR may be the way to go.

Redeployable packaging?

Erlang is packaged for win, linux, and mac. CEAN (Comprehensive Erlang Application …) ... Wings. Martin Logan, faxian. Martin Logan also has generator applications for Erlang (Erlware?).

Good frameworks for doing ontological modeling and reasoning?

Expert systems shell. University of Corona. Reported at the Erlang user conferences. Multi agent listener but not generalized to owl.

List of reference projects?

No canonical reference. But check trapexit.org.

Where is Erlang center of gravity?

Attracting lots of interest in Financials – especially in London. Finance industry targeted conference coming up in London. Much bank interest – extreme real time demands. Number of pure erlang based trading companies. Debt buying company – purchases debt and collect. Buys these debts a few moments after the transaction. Mnesia founder involved.

Future changes – where is it going?

Not many changes to the language. Millions of legacy lines of code needed for compatibility – syntax unlikely to change. But, implementation speed optimization and multi-processor support is likely to improve.

Embedded Erlang?

4MB probably the minimum size. Used to be 640k but may be a better domain for C.

String localization – UTF8?

In a sense it is solved. In another sense it is not solved. In Erlang there is no string type – it is a list of integers. UTF8 is an integer and can be stored. However, the interpreter from text to binary and back is not written. However, it is not difficult. It is in fact a library change.

What is a good candidate for Erlang?

Telecomm applications, lots of small processes. Individual processes have low ammount of short computations. Good at coordination, concurrency, switching. Not good at matrix multiplication, gif encoding etc. Financial applications fit the erlang profile.

Libraries for interfacing to SQL databases?

ODBC libraries. Could just open a socket to the DB. Java has a good arbitrary support for sockets. Check Yariv’s Erly web implementation.

Summary

Fault tolerance drives a lot of the design decisions behind Erlang. Simple functional language. Function selection is by pattern matching Variables are immutable

Concurrent Erlang

3 primitives: Spawn, Send, Receive – very simplified from Object Oriented model.
register / unregister can be used to associate a name with a process

Fault Tolerant

catch .. throw, try … catch … end
link, process_flag(trap_exit, true)

Distributed Erlang

spawn(Node, Mod, Func, Args)
or explicit term passing

Benefits of Erlang:

Technologies becoming more concurrent and people are looking for solutions to that probelm. Cooperative applications, etc. Erlang is a working solution. Not operating system dependent. 20 – 30 years of experience with fault tolerance measured in years.
Multi-Core ready
Processes in the language

Some projects to research:

AXD301 – biggest Erlang and Functional programming ever made 60 programmers x 3 years. Runs backbone of british telecom. Market leading in that sector. Peaked at 9×9s of reliability. This wasn’t a commercial project on 2-3% of the market.
Kreditor (Kreditor.se) – buys debt. #2 IP startup in sweden. $150 year. Self funded. Founded by alumni of Blue Tail.
SimpleDB (Amazon)
CouchDB (text db)
MociWeb (Mochimedia)
Ejabberd (jabber server – by extension Twitter?)
ErlyWeb (Erlang web framework inspired by Rails)

Erlang Pragmatic Studio - Day 2 Notes

2008-02-14T19:07:00Z

Erlang Pragmatic Studio with Joe Armstrong & Dave Thomas - Chicago, Feb 13-25, 2008. h2. Day 2 - Feb 14,2008. Here are my notes from Day 2 of the Pragmatic Studio Erlang course. Note that these notes are only vaguely edited. Be aware that these are notes so there are certainly typos and errors. h2. Concurrency review of an implementation of map:

map(F,[]) -> [];
map(F,[H|T]) ->
	[F(H) | map(F,T)].

Erlang is a concurrent language and a functional programming language second. Why concurrent? The world is concurent - external world & computing world. Often you inherit all the limits of the operating system in regards to processes. Erlang is not bound by the operating system. The emulator handles concurrency. * Programming concurrent activities in sequential languages is artifically difficult. * Shared memory (locks, mutexes, coroutines, processes, thread, deadlock, livelock, failure, thread-safe) * Message passing Web servers must spawn a new thread for every request. In Erlang when one thread crashes, only that one crashes not the whole system. This is not the case in other environments. Isolation of components is the key to the pattern. Dave: * process = object * message = method * can apply the same techniques in OOD. The small talk paradigm for passing messages unifies this (to a certain extent). Joe: Alan Kay in his origination of object orientation, messaging was more primary. The problem with a function call is that the message must return to the caller. Concurrency in other languages * can only create very small number of processes * heavy weight * only supports message passing and not the kind of error handling semantics that Erlang has If you know the name of a process, you can send it a message: * Pid ! Message * If Pid is hidden you cannot send a message to the process Message arrives in a mailbox (like email, can be stacked). You never know if a message arrives (if you want to be sure send a message back). Only three primitives for turning sequential program in to concurrent code: * spawn - create new process * send * receive - try to match pattern h2. Spawn Three syntaxes: * spawn(fun foo/0) - runs foo/0 in a new process * spawn(fun() -> ... end) - spawns and inline fun * spawn(Mod, Func, [Arg1, Arg2, Argm]) performs apply(Mod, Fun, [Arg1, Arg2, ..., Argn]) in a new process. In Pid1 evaluate Pid2 = spawn(fun() -> ... end) h2. Send * Pid ! Msg * returns the message (e.g. A ! B ! C ! Msg) h2. Receive

same as a case statement:
receive ->
	pattern
	...
end

self() is the pid of the current process. including it in a message allows the recipient to respond to us.

% receive whatever message comes first
receive
	Msg ->
	...
end

% receive messages in order

receive
	foo -> true
end,
receive
	bar -> true % leave bar in the mailbox until foo is received
end

B!{transfer,self()}

receive
	{transfer, A} ->
		C ! {transfer, A}
	end
	
receive
...

promise and yield are non-primitive abstractions. Are we hiding the wrong thing? Should we expose the protocol? Pros * Hides message structure * Makes module reusability easy * we know how to define APIs Cons * More code * We don't know how to describe protocols * Protocols do not exist in Erlang as first class objects Key factor minimizing cross machine boundary cost. Abstracting server functionality * run a server that can run in the background * needs some kind of state * pass in state at the start * in case of counter pass in 0 * then cast Mutate state through recursion. Two more items. Registered processes. * Any program that wants to send a message to a Pid must have Pid in a local variable. * Use register / unregister * register(name, Pid), name ! Term sens a message to Pid * whereis(Name) returns Pid or undefined * mind the propigation delay Example of register: register(counter, spawn(fun() -> counter(1)end)). From shell see all the registered processor with: regs(). Registered names are global resources. If two processes register with the same name one will fail. receive... after wait for a certain period of time - timeouts

receive
	pattern1 -> ...
	pattern2 -> ...
after
	TimeInMilliseconds -> %can be atom {infinity}
		Actions
end.

Pid = spawn(fun() -> wait() end),
Pid ! {do, F}

wait() ->
	receive
		{do,F} ->
			F()
	end.

Security risks. SMP and Processes * Erlang can take advantage of multiple processes. * Enable using -smp +S n options only if compiled with correct flags all concurrent behaviour is programmed with the following (instead of blocks, mutexes, threads, etc...) * spawn * send * receive * default max processes is 327xx * increase processes with: erl +P 1000000000 * once it passes processes:max(1).

MACRO: Debugging
-define(DEBUGGING,true).
-ifdef(DEBUGGING).
-define(DEBUG(X),X).
-else
-define(DEBUG(X),void).
-endif

?DEBUG(io:format("Found:~p~n"),[Files]),
...

Two ways to do distributed computation: * distributed Erlang as cluster with one name space. Consider different machines may have different versions of Erlang. Getting the right code onto the right machines may also cause issues. Fully connected machines don't scale well - > 90 machines gets difficult. * Explicit sockets. No issues of ownership. More easily scalable as other issues are encapsulated - memory is fully not shared. h2. Client

{ok, Socket} = gen_tcp:connect(Host, Port, [Options]),
	ok = gen_tcp:send(Socket,Data),
	...
receive
	{tcp, Socket, Data} ->
		%% do something with the data
		...
	{tcp_closed, Socket} ->
		%% take care of this ...

How does TCP work? * server needs to be able to hand multiple clients trying to connect to it. The protocol is slightly different. * c level api - listen(ip, port); all that listen does is say I will support people on this given port. call accept and turn it into an IO object that you can use. Typically each connection spawns a different process. a single-shot server accepts one connection: start_server()

{ok, Listen} = gen_tcp:listen(Port, ...)
spawn(fun() -> par_loop(Listen) end).

par_loop(Listen) ->
	{ok, Socket} = gen_tcp:accept(Listen),
	spawn(fun() -> par_loop(Listen) end),
	loop(Socket).

Packet lengths: * A 4 byte length header is automatically added/removed by the system when calling gen_tcp:send and messages are assembled to the correct length before {tcp, Sock, Data} message are sent to the controlling process.

gen_tcp:connect(Host, Port, [..., {packet,4, ...}])
gen_tcp:listen ....

Sending Erlang terms
gen_tcp:send(Socket, term_to_binary(Term))...

receive
	{tcp, Socket, Data} ->
		Term = binary_to_term(Data),
		...

The Middle Man Pattern: - Middle man has to be written for a particular protocol. - like unix filters (Dave) - piping

loop(Pid,Socket) ->
	receive
		{Pid,Msg} ->
			gen_tcp:send(Socket, term_to_binary(Msg)),
			loop(Pid, Socket);
		{tcp, Socket, Data} ->
			Pid ! binary_to_term(Data),
			loop(Pid, Socket);
		{'Exit',Pid} ->
			gen_tcp:close(Socket);
		{tcp_closed, Socket} ->
			exit(Pid, socket_closed)
		end.

use term_to_binary and binary_to_term. when to use: * , separate individual expressions (smallest scope) * ; pattern matching * . at end of function (biggest scope) normally you don't parallelize file read/write access - another case of shared memory.

% from simple_coordinate_server.erl
% pass in milliseconds or infinity to sleep forever
sleep(T) ->
    receive
    after T ->
	    void 
    end.

gen_txp:listen(0 ...) % gets a free port

Concurrent patterns: * client server model is the most used pattern when writing concurrent programs. * can also use worker manager pattern. * or finite state machine. OTP course would require Erlang background.

try
...
catch
	exit:Why ->
		From ! {Name, die, why}, % die and tell calling process
		loop(State)
end

delegation is a nice place to do load balancing: client, delegator, responder Erlang was developed in a trusted environment, many of the techniques are applicable to this environment. This is an empty server that will run anything passed to it (guest account slightly safer but still incredibly permissive):

Pid = spawn(fun() -> wait() end),
...

Pid ! {do, F}
...
wait() ->
	receive
		{do, F} ->
			F()
		end

Many variations of concurrency: * Resource managing * Mobile Code * Mobile code with transactions Systems can be specified functionally. * Systems have non-functional behavior (scalability, fault tolerance) - usually defined very weakly. * These are usually vaguely defined. * In Erlang these things can be specified clearly. In Erlang you often write functional and non-functional code separately. These can then be used independently. Can abstract things like scalable fault-server and fault tolerant processing. Cool! You can say how many failed processes are allowable in a given period. This can be specified programatically or in product requirements. e.g. if there are three machines and one machine fails and data is lost it is warranted, but if two machines die and data is lost it is out of warranty. h2. links links define error propagation paths If A is linked to B then: * If A fails then B will be notified * If B fails then A will be notified Signals and Messages: * messages are sent with send * signals are sent when processes die. Signals are not messages. * if process A is linked to B it A will be sent an exit signal if B dies. layers to make the system fault tolerant: * application level * level 2 - trap exits * level 1 - core level to trap exits considerations: * if you do spawn followed by link the process might die before you get to the link statement * spawn_link is like spawn followed by link only the two are performed atomically. * trap_exit principles of remote error handling * fault tolerant systems need two separate computers * let one process do the work * let some other process fix the error, even remotely. If a whole machine is crashed it must be fixed remotely. This pattern should also be applied on a local system - cross machine boundary should be handled in the same manner for scalability. * exit signals are like uncaught exceptions that escape from the process * failing processes should fail early How do you know when a machine fails? In distributed Erlang, there should be a heartbeat process that is sent between machines (1-10 times a second). use halflink for monitoring. Link methods: * link * unlink * process_flag(trap_exit, true) - tap exits * {'EXIT',Pid, Why} - message sent when a process dies These methods are orthogonal to spawn, send, receive

Erlang Pragmatic Studio - Day 1 Notes

2008-02-14T02:26:00Z

Erlang Joe Armstrong & Dave Thomas Chicago, Feb 13-25, 2008

Here are my notes from Day 1 of the Pragmatic Studio Erlang course. Note that these notes are only vaguely edited. Be aware that these are notes so there are certainly typos and errors.

Overview

“First public Erlang course in US.” Joe invented Erlang in 1986.

Fault tolerant systm:

Two computers, no shared data. Needs to be isolated computation.
Concurrent & distributed by default.
Then becomes scalable. Not sharing data key for fault tolerance.

Next to tackle – low power computing. Lots to tackle. Computing and MIDI messaging interest.

Uses:

Faster rules engine. Better distributed system than Java. Distributed locking across nodes is really error prone.
P2P. Center server is used to coordinate edge – erlang.
Internet worm emulator. Evaluate security.

Functional programming overlap with Ocaml, Haskel, Erlang family. Central idea of non-mutable state is critical to all functional languages – and in stark contrast to object oriented programming.

Day 1 – Sequential programming
Day 2 – Parallel programming
Day 3 – Distributed programming and OTP

Pairing suggested for the labs. One core concept for the labs.

Write a simple distributed client-server.
Write a DNS like server resolver.
Write a universal server and register it with DNS.
Send code for a web server to the universal server and tell it to execute it (mobile code)
Correct an error in the code and fix the error without stopping the server

Going from one to many machines can take hours to solve (e.g. sorting out why packets aren’t making it through firewalls).

The erl shell

Erlang shell: More a controlling environment than it is a full fledged interpreter (in comparison to say ruby’s IRB). How to get out:

unix ^{\ aborts everything}
G => submode => press q
^C – a for abort

When you type ^G it’s like ssh:

Can jump to different shells. It’s distributed.
type h for help

Shell has standard keybindings – has emacs keybindings as well.

Data Types and Assignment

Integer arithmetic in Erlang is very good. However, if you want extremely fast integer arithmetic you would writ it in C.

X = 1.

Works differently than object oriented languages. Once value is assigned it cannot be changed. Variables are unbound or bound. Thread safety is anathema to Erlang. This is because there is no mutable state and semiphores are not necessary for protecting them.

In Erlang X=1 is a pattern matching in operator (not traditional assignment).

”=” is a very conventionally overloaded opperator. = was chosen for ease of use, but it is not used as it is in other languages.

Worker supervisor model of programming. Assign process to do something. If it fails get some other process to do it. Likewise if pattern fails, it just throws an error. Not like the constant error checking in mutable state applications.

Note: pattern matching errors in Erlang 5.6 are nicer than in 5.5.

Variables uppercase: X, Y, Bob

Numbers:

Integers – 1 34
Base – 2 2#101010
Base – 8 8#0723
Floats – 12.34e-07

Atoms:

lowercase
true false hellow_world
they evaluate to themselves

In Erlang everything is a 32 bit or 64 bit word. Atoms are stored once in memory (like ruby symbols). For comparison:

X = joe.
Y = joe.
X = Y. %These point to the same memory location

Tuples:

{food, egg}
can be nested {person, {firstname, “joe”},{lastname,”armstrong”}}

Bind multiple variables: {A, B} = {1,2}.

In a tuple, there can be multiple occurrences of a variable. This throws an error since C will be bound to 1. C will still be unbound – the whole thing fails: {C, D, C } = {1, 2, 3}.

Atoms can also be matched: {person, Name} = {person, “Dave”}.

f(). %forget all bound variables

Note: you may get an error in the book if you already have a variable bound and attempt to reuse it in another example.

h2. Lists List are used in 95% of your functions.

[1,2,3,4]
[] %the empty list

A = [1,2,3].
[C, D, E] = A.

Lists are a naturally recursive data structure. Lists can be expressed as a head and a tail: [H | T]

Head is something, tail is always a list. In C this would be a linked list. This notation originated in Prolog. Much of Erlang originated in Prolog – in fact the earliest versions of Erlang were implemented in Prolog.

Why Lists and Tuples? (these are the glue in the language):

Lists are for variable number of things. Lists are traversed and the first element is the only one that can be efficiently accessed through head and tail notation.
Tuples are a container for a fixed number of things.

Construct a new list (on the right hand side of the operator it’s constructor):

A = [1,2,3].
B = [ fred | A].

Deconstruct a list using pattern matching (on the left hand side of the operator it’s a deconstructor):

[ H | T ] = A.
H = 1.
T = [2,3].

Head of the list could have multiple values: [ H1, H2 | T] = [1,2,3].

Compiler optimizes your code so that it is only patterns. If and case expressions are eliminated.

h2. Strings

Strings are actually integers:
A = "Dave".
[H | T] = A.
H = 68.

Lists of numbers will be returned as characters:

[65, 66, 67].
=> "ABC"

”_” matches anything and doesn’t bind:

A = {name, "Dave", state, "TX"}.
{ name, Name, _, _} = A.

++ concatenates lists.

Interesting list length counts:

A = "hi,".
B = "there".
length([A,B]).
=> 2
length(A ++ B).
=> 8
length([A | B]).
=> 6
[A | B]
=> ["hi,",116,104,101,114,101] %"hi" is a nested list

Modules

Useful shell commands:

help(). %show all shell commands
b(). %display all bindings
v(N) %value of a query. Type the number of the prompt or -X and it will return the value at that point in your program.

creating a module:

-module(dave).
-compile(export_all). %export all of the

Functions are defined by their name and “arity” (number of arguments that they take). Semicolons separate pattern matches and the functionality that should be applied. In erlang there is a tendency to write patterns rather than to write conditional loops.

Example of sum function with a test:

-module(dave).
-compile(export_all).

test() ->
    6 = sum([1,2,3]),
    10 = sum([1,2,3,4]),
    0 = sum([]),
    yes.

sum([H | T]) ->
    H + sum(T);
sum([]) ->
    0.

Why doesn’t erlang have a loop?

Joe: you don’t need it.
Dave: A loop usually includes a variable with state. Implicitly this has mutable variables.

Could write a test that runs all test methods found in each module and make sure final atom “hooray” or “yes” is returned.

Erlang idiom is not to do defensive error case checks. Program for the happy case in Erlang. Then the question is what are you going to do with errors?

The set of things that can happen outside of the specifications is nearly infinite. In Erlang only code that is specified and nothing else. If it goes outside, let it crash. This is very different than defensive programming.

Let it crash.
Crash early. If it crashes early it will be available in the logs and the state will be traceable.

On shipping, wrap the errors.

In a concurrent world there are many more things that can go wrong and are not reproducible. Joe’s thesis was on postmortem analysis of bugs.

%%File name is amod.erl
-module(amod).
%%these can be called from outside the module
-export([foo/1,bar/2]).

compile from command line:

erlc amodule.erl
=> outputs amodule.beam

run it: erl -noshell -pa /path/to/code -s Mod Func Args

erl has about 60 flags.

erl scripting

escript:
/#!/opt/local/bin/escript
main(List) -> %%main is first function called?
    ...

Code paths etc:

./erlang or ${HOME}/.erlang
code:add_patha("/path/").

loads using undefined function – done using an exception. Tail recursive variable that carries through the state. Equivalent of System.out.printline:

    io:format("Hello from: ~p~n",[:file:get_cwd()]).

If you installed from macports you will have the man page. If you compile you will have to add them in explicitly. View man pages like this:

erl -man io

Erlang is also on the website. May be faster than local man pages. erlang.org/doc/man

datatypes:

integers
floats
atoms

structured datatypes:

lists
tuples

Functions

”->” translates to “returns”.

BIF means “Built in Function”. Allows you to get out of Erlang and closer to the metal. These are not written in Erlang. Can see them in:

erl -man erlang
or in the book

Any term can be serialized into binary. Joe: This is very optomized, a highlight of the language: gen_tcp:send(Socket, term_to_binary(Term)), ...

receive
    {tcp_data, Socket, Bin} ->
        Term = binary_to_term(Bin)
end.

tuple_to_list sometimes erlang:now(). In theory erlang: BIFs are implemented in erlang. But it turns out that this is not completely true. This is quirky.

Select a function to apply (Erlang meta-programming anyone?):

apply(Mod, Func, [Arg1, Arg2, ...]). %this is like a ruby send
M = list_to_atom("erlang"). %string is in fact a list

Funs, a bit like calling a lambda:

Double = fun(X) -> 2*X end. %define a function and then use it as a parameter!
lists:map(Double,[1,2,3,4])
lists:map(fun(X) -> 2*X end, [1,2,3,4]). %more idiomatic

There is no currying.

Writing your own control structures is common. For is not built into the language. See Joe’s Erlang book.

Funs can return funs:

MakeAdder = fun(Inc) -> fun(X) -> X + Inc end end. %this is a closure
Add5 = MakeAdder(5).
Add5(10).

Can write programs that only generate closures and evaluate them later (lazy evaluation). This is a technique that haskell programmers often use.

Some of the most useful functions are:

map – takes a list produces a list of values.
fold – takes a list and produces a single value.

lists:map(fun(X)-> X* X end, [1,2,3]).
=> [1,4,9]

Joe calls this list at a time programming. Makes programs much shorter.

foldl

applies a function to every element in a list, updating an accumulator as it goes. Returns a single value (the accumulator).
lists:foldl(Fun, Acc0, List)
great for summing

write sum using an accumulator (may be a bug/typo here):

sum(L) -> sum(L,0).

sum([H|T], Sum) ->
    Sum1 = Sum + H, %tail recursion is applied to Sum1 since it is called once and never accessed again & garbage collected.
    Sum(T, Sum1);
sum([], Sum) ->
    Sum. %at the end just return the accumulation value

Custom folds are extremely useful….

List Comprehensions:

come from set theory
eg {x : x is even} – the set of all even numbers
in erlang the colon is written as ||
[X || X <- L, (X rem 2) =:= 0]. % same as {x : x is even}
the above should be more efficient than map – compiler is optimized for list comprehensions.

[ Constructor || Pattern <- List, Predicate, ... ]

Generate the cartesian project:

[ {X, Y} || X <-><->

The quicksort list comprehension:

-module(qsort).
-export([sort/1]).

sort([Pivot]|T) -> %arbitrarily choose first element as pivot
  sort([X || X <- t>< Pivot])
    ++[Pivot]++
    sort([X || X <- t>= Pivot]);
sort([]) ->
    [].

Note that the quicksort function in Erlang actually is quite a bit longer than this and handles lost of special cases.

Accumulators pass extra argument(s) into the function:

-module(accum).

evens_and_odds(L) -> evens_and_odds(L,[],[]).

evens_and_odds([H|T,E,O]) when H rem 2 =:= 0 -> evens_and_odds(T, [H|E], 0); %gaurd in the first clause
evens_and_odds([H|T, E, O]) -> evens_and_odds(T,E,[H | O]); %no gaurd in the second clause
evens_and_odds([], E, O) -> {E, O}.

Adding gaurds allows the compiler to compile better. Erlang has multiple entry points to the function unlike Java and C. If you write your code without multiple entry points and with an if statement, you don’t have the opportunity to reorder the clauses. If you write this with an if statement you have done the compilers job. Better not to.

There are a few equality operators. Only different when comparing floating point numbers:

* == %fuzzy equal for integers and floats comparison
* =:=

Can’t call functions in gaurds. Know that they can be parsed into optimal decision trees.

Single Gaurd
Full Guard – comma seperated list of single guard conditions.

Gaurds sequence:

Commas are interpreted as and.
Semicolons are interpreted as or.

It’s messy to write a new function for every condition. This is reduced by writing guards:

case and if

%case statement
goto_work(Day)->
    case classify(Day) of
        weekday -> true; %could add a gaurd after weekday
        weekend -> false
    end.

goto_work(Day) ->
    if
        Day =:= monday -> true;
        Day =:= tuesday -> false;
        true -> false
    end.

compiler reduces case and if to:

f(pattern1) ->
    action;
f(pattern2) ->
    action2.

Booleans:

true and false are atoms
these are by convention only – but this is strongly recommended convention

Given a temperature that returns hot or cold. It is better to use is_hot and is_cold and return true and false. This will allow you to use functions that work with booleans. e.g. lists:filter(Fun, L) -> L’

Tame Large Tuples with Records. These expand into tuples. Joe thinks this syntax may improve (e.g. he says “it sucks”).

Example:

%Define a record - module syntax only. Won't work on the command line.
-record(person, {name, firstname, lastname, age, sex, weight, height})

X = #person{name="jane",age=32, sex=female} %%creates a new record
birthday(#person{age=N} = X) ->
    X#person{age = N+1}

There are quite a few functions that return tuples that are generated by records. These are records that are returned as tuple. If you include a specific header file it will be interpreted as a record.

It’s better not to use records for short tuples. This is the idiom.

Not mentioned here:

macros – first introduced for language customization (internationalization), then grew a bit in scope.
parse transforms – for implementing extensions to the language

Exceptions.

Overview:

exceptions are generated when errors occur
exceptions can be generated by a program: throw, exit, error.
convert exceptions into values

X=atom_to_list(123).
error...
X. % is unbound

catch F(X) returns F(X) if no exception is generated while evaluating the function. if the error is thrown returns a tuple.

some_function(X, Y) ->
  case something_that_might_crash(X,Y) of
    {'EXIT',Why} ->
        %% handle error;

... Or try catch. Gives more fine grained control.

start(Arg1, Arg2) ->
    try
        begin
            ok = start_server(),
            L1 = dothis(Arg1),
        end
    end
    catch
        exit: Pattern ->
        ...
    after
        ...
    end.

Abnormal termination with throw:

..
    catch
        throw:X ->
            X
    end.

Use exceptions for “impossible” situations – thinking defensively goes awry. Everything has return value so the above actually returns the tuple {ok}. This takes a correct crash and returns something incorrect:

opcode(load) -> 1;
opcode(store) -> 2;
opcode(X) -> %%invalid: what I do now?
    io:format("bad opcode:~p~n",[X]).

a “correct solution” to the above:

opcode(load) -> 1;
opcode(store) -> 2;
opcode(X) -> exit({ebadOpCode, X}).

Solution two: do nothing (best):

opcode(load) -> 1;
opcode(store) -> 2.

Turn values into exceptions:

exitIfCannotOpenFile(File) ->
    case file:open(File, [read]) of
        {ok, Pid} ->
            Pid;
        {error, Why} ->
            exit(Why)
    end.

Improving and error:

readConfigFile ->
    case file:consult(File) of
        {ok, Stuff} -> Stuff;
        {error, enoent} -> exit({eBadConfigFile, File})
    end.

Three ways of generating exceptions:

exit – punish the programmer
throw
error

Alternative to defensive programming – one big catch at the top of the program. Crash early, crash often.

Main sequential libraries:

lists.erl – list manipulation. Extremely common, should be familar with these. There are lots of these. Joe says about 15 are in common use, then diminishing returns after that.
file.erl
ets.erl – Hash tables in memory.
dets.erl – Persistent hash tables on disk
dict.erl – fast key-value dictionary
io.erl and io_lib.erl
filename.erl
filelib.erl
file.erl

Above the BIF level you can actually read the source code of the lbiraries

find using code:which(lists)
mac location of libraries (for maports install)/opt/local/lib/erlang/lib/stdlib-1.15
have a look at /opt/local/lib/erlang/lib/stdlib-1.15/src

Three common patterns

Pattern 1 – Terminate after value is found

do(..., [H,T], ...)     -> ...;     % found pattern, terminate recursion return value
do(...,[_|T], ...)     -> do(...,T);     % pattern not found, pass in tail and recurse further
do(..., [], ...) -> ... .             % end of recursion return value

example of pattern 1:

member(H, [H|T]) -> true;
member(H, [_|T]) -> member(H,T);
member(H, []) -> false.

Pattern 2 – Build lists as the output of a recursion.

filter(Fun, [H|T]) ->     %grab a function and a list split into head and tail
    case Fun(H) of        %evaluate function on head
        true -> [H|filter(Fun,T)];    % if true, return append head to the filter called on tail
        false -> filter(Fun, T)        % if false, return apply filter to tail without appending
    end;
filter(Fun,[]) -> []. % if passed the empty set return the empty set

Pattern 3 – Build results in reverse order in an accumulator

partition(Fun, L) -> partition(Fun,L,[],[]). %delegate public function to another function

partition(Fun,[H|T],Yes,No) -> %take a list of a function, split list, Yes list, and No list
    case Fun(H) of               %apply function
        true     -> partition(Fun,T,[H | Yes], No); %if true recurse Fun, Tail, Add head to Yes, and No.
        false     -> partition(Fun,T,Yes, [H | No]) %if false recurse Fun, Tail, Yes, Add head to No.
    end;
partition(Fun,[],Yes,No)-> 
    {reverse(Yes),reverse{No}}. %if list is empty pass back the reversed list

Methods of reversing a list:

[H | Yes] %adding and needs reverse
Yes ++ [H] %right order but slower than reversing at the end

Remember, all of these can be done with pattern matching:

validation
deconstructing lists
case statements
list comprehensions

They all use this one form:

f(Pattern1) ->
    action1;
f(Pattern2) ->
    action2.

Build Ruby Audio APIs Hack Session Notes

2008-02-04T17:47:00Z

Thursday Jan 24th 2008 the Bay Area Computer Music Technology Group met for our "Build Ruby Audio APIs Hack Session at Internaut Design.":http://electronicmusic.meetup.com/152/calendar/6806947/ Thanks to David Lowenfels at Internaut for hosting! Our focus was identifying existing Ruby Audio APIs for extension, and determining what would be the most useful approaches and protocols. We identified four layers to explore and technologies that could be extended or bridged for each layer: * Raw DSP * Sound design * Composition layer * Inter-application communication The first three layers essentially map to timescale and the fourth to inter-application protocols. The programatic structures would however be different for each of these time scales. h2. DSP Layer For raw DSP there was an agreement that Ruby interpreters would not be directly fast enough to run serious DSP code and would likely need to rely on C libraries. We discussed generating csound orchestra files using erb and wrapping the csoundapi. I have been making a preliminary pass at writing a csound orchestra file writer that uses ERB that I will be posting to a public subversion repository in the near future. David Lowenfels was interested in writing DSP code directly in Ruby and did some further inquiry into SND/Ruby that seemed very promising see the email below. The other alternative was directly accessing portaudio through ruby. A wrapper for this already exists but doing DSP this way seemed unlikely to run quickly or be a truly productive line of inquiry. There was some speculation on whether a Ruby interpreter would ever be fast enough to run DSP code but we came to no conclusion on this. Currently it looks like YARV is the fastest interpreter (needs verification). SWIG was mentioned in a number of contexts as being an excellent way to bridge C libraries to Ruby (as well as an array of other languages). h2. Sound Design Layer VST and Audio unit wrappers could presumably be easily written. Tim Thompson's experience writing a Python API for VST was that working with VST would be succesful and strightforward. Audio units are probably well documented. We didn't explore too far into this territory. Using ruby in Max/MSP through jRuby and the JVM seems possible and straightforward if someone has the will to document the setup this could be fairly widely used. I am pretty sure it could be successful as I have used Jython with Max/MSP and it worked. There was also a suggestion to use jRuby and jMax, but extending max/msp elicited more enthusiasm and would potentially be of wider use due to installed user base. Wrappers for csound would also be useful for sound design. h2. Composition Layer Ruby is an excellent base language to extend for Domain Specific Languages (e.g. rails). A composition framework or DSL could be written on top of any of the technologies discussed for the DSP or Sound Design Layer. However, the most interesting approach (for me anyway) would be writing the composition layer to talk to OSC, thus providing the most portability across audio engines. h2. Inter-application communication Making use of the Ruby OSC wrapper seemed like a very productive path. We discussed using OSC to expose the object space of a system. We also decided it would be useful to perform some stress testing for the existing Ruby OSC library (rosc) and optomize OSC library if necessary. OSC would provide interactions with the most platforms of interest and would allow the creation of a more high level API or domain specific language written in Ruby. h2. Future Events At the end of the evening we reviewed the hack session itself. From this discussion it is likely that future hack sessions for the Bay Area Computer Music Technology Group will be organized around OSC and interactions with multiple languages including Ruby. It also seems that a more general Audio Open Source development hack session would be of interest. There may also be more Ruby focused hack sessions, or a Ruby focused sub group within a larger hack session. h2. Useful Links Wrapping Csound: * "Interesting Csound API info here":http://lac.zkm.de/2006/papers/lac2006_victor_lazzarini.pdf (PDF) Using the Portaudio Ruby wrapper: * "Wrapper":http://samesimilar.textdriven.com/ * "Portaudio Site":http://www.portaudio.com (was down last time I checked) Using OSC from Ruby to control audio applications: * "An existing ruby OSC library - ROSC":http://hans.fugal.net/src/rosc/) SNDs Ruby functionality: * "About SND":http://ccrma.stanford.edu/software/snd/snd/snd.html * "SND & Ruby":http://ccrma.stanford.edu/software/snd/snd/grfsnd.html#sndandruby h2. Further Developments David post meeting interchange about SND/Ruby: seems like my intended work with ruby-clm is basically already done! :) I'll report back when I have generated some audio examples. -David Begin forwarded message: > Hi, David! Thanks for your interest in Snd/Ruby. > >> I'm excited to see that I could generate audio with ruby. How does >> this fit in with Snd? > > If you configure Snd with ./configure --with-ruby, you will end up > with a new Snd program using extension language Ruby. You can type in > Ruby/CLM/Snd commands in the Snd listener, eg: > > x = 1 + 2 > os = make_oscil > os.call(0, 0) > os.frequency = 330 > > etc. There exists a bunch of *.rb files which are mainly translations > of the corresponding Scheme files. Snd's documentation tries to show > the similarities between Scheme and Ruby Snd/CLM code. > >> Is it possible to run ruby-clm outside of snd? > > Get the file ftp://ccrma-ftp.stanford.edu/pub/Lisp/sndlib.tar.gz and > configure it with ./configure --with-ruby. If you install sndlib.so > in ${RUBY_SITE_ARCHLIB} you will be able to require it like all other > libraries with 'require "sndlib"'. The contents of sndlib is nearly > the same like Lisp CLM which is described in snd/sndclm.html. A few > *.rb files are included in sndlib.tar.gz which may help a little. > > If you have more question please don't hesitate to ask them by email, > or even through the clm mailing list. > > Mike

ChucK Workshop With Ge Wang @ CCRMA - Notes!

2008-01-18T22:07:00Z

Below are the notes that I took at the ChucK audio programming workshop lead by Ge Wang at Stanford CCRMA on Sunday Jan 13, 2008. Ge Wang and I organized this event through the Bay Area Computer Music Technology group. Many thanks to Ge for teaching this! You can join the Bay Area Computer Music Technology group if you would like to be informed of the advanced workshop, which should happen in another few months.

These are raw notes. If you notice any egregious typos let me know and I will correct them. Happy ChucKing…

ChucK Workshop Notes – Lead by Ge Wang – Sun, Jan 13th, 2008

Code == musical instrument On the fly programming

Audicle is fun and crazy to use, but it’s still a line of research. MiniAudicle is more usable.

Language design solutions:

Make time itself computable
Synchronize events
Flexible and Readable – optimize later

The CHUCK Operator

this is the chuck operator:

=>

simple chuck:

x => y;

action depends on what x and y are. chain chuck:

w => x => y => z;

operations run from left to right. This is backwards from other langs like (c/c++, Java) nested (do what’s in brackets first):

w => (x => y) => z

unchuck:

x =< y =< z;

feedback:

x => y => x

Time and Basic Usage

dur is a native type

Basic code test:

SinOsc foo => dac;

Generate a sine wave. foo instanciates SineOsc. foo is like a patchcord.

440 => foo.freq;

To make sound you must advance time:

1::second => now;

comments:

//this is a comment - same as Java or C/C++
/* can also use c style comments */

There are emacs keybindings for chuck. Others? Coloring in MiniAudicle.

default intensity is 1. chuck will not advance time for you. you must advance time.

basic program:

//----------------
SinOsc foo => dac;
.5 => foo.gain;

//advance time!
220 => foo.freq;
.5::second => now;

windows and linux versions work. Mac OS X is the easiest to use.

MiniAudicle:

start virtual machine
add shread to run. can add multiple threads.
replace thread, replaces old thread code, replace just replaces the latest thread
remove thread removes thread on top
watch for errors in the console monitor

for os x users only go to

miniAudicle => prefereces
enable on-the-fly programming visual feedback (show you gostbusters x when there is an error)

random number program:

SinOsc foo => dac;
.5 => foo.gain;

//advance time!
while ( true )
{
    //choose frequency randomly
    Std.rand2f(30,1000) => foo.freq;
    100::ms => now;
}

consider:

3/2 does integer division not floating point division
3/2 = 1
3./2 or 3.0/2 = 1.5

100::ms => now; //does not use system clock
//puts code processing to sleep while stepping forward

if code needed to advance time is not computable in the given time you will have dropouts you can compute in “real time” or “off line”

can inject code into running threads, but this is programmatic rather than a built in part of the virtual machine.

time constructs:

1::week //stopped here because month and week are variable size
1::day
1::hour
1::minute
1::second
1::ms
1::samp

could do:

52::week => dur year; //define 1::year

up next: sounds, parallel, controllers

SinOsc foo => dac; //foo is local to this shred

Links on the ChucK Language

home page for ChucK is: chuck.cs.princeton.edu – download chuck, miniAudicle
language specs

Check out the unit generators listing in the specs. Eventually users will be able to develop their own unit generators.

BiQuad is all you need to implement a filter of any complexity (due to re-factoring). ResonZ is (probably) the same as reson~ in max/msp

STK unit generators are in chuck: Envelope, ADSR, Delay, etc. STK instruments unit generators BandedWG – tibetan bowls, and bowed instruments Bowed – cool but not realistic Flute – excellent model but hard to control (lots of params) Shakers – awesome, 22 presets maracas to points in (statistical method for modeling particle systems – from Peri Cooks Physm)

Small Musically Expressive Laptop Toolkit

has some great chuck code
keyboard and mouse events

More Complex Sounds (than a sine wave)

Basic Harpsichord sounding instrument with verb code:

StifKarp foo => JCRev r => dac;
.1 => r.mix;  //set reverb ammount 0 is none, generally params are 0 -1

//advance time!
while ( true )
{
    //choose frequency randomly
    Std.rand2(30,90) => Std.mtof => foo.freq;

    //note on!
    1 => foo.noteOn;
    //advance time
    500::ms => now;
}

There are 50,000 line chuck apps out there!

Ge’s Q&A:

Object system is still not fully implemented.
Laptop orchestra recordings are on line (but this doesn’t give 90 discrete channels of sound that you would get at a concert) – good chuck examples
There are some good non-PLork chuck pieces as well
Wiki and forum also has chuck code.
Piece called “TBA” – first instance of orchestral livecoding
Coding as a gesture. What are the aesthetic of laptop performance beyond the glow of the laptop on the face.
The score as a page of code that is going to be typed…
Livecoding communication with comments
Rehearsal becomes important… knowing what kind of sounds you can get out of your algorithms becomes tremendously important.

Synchronization

There is a rudimentary include system in chuck. Somewhat goofy at this point. Can do code injection on a running process.

Chuck model, no control rate. There is only the advancement of time. Can create your own control signals if desired (e.g. Math.sine )

By default most oscilators have only one input for audio, but have many inputs for control. run three threads on the command line

Time based concurrency: > chuck moe larry curly

If they are started concurrently they are all controlled off of the same clock and time is advanced in a concurrent fashion.

By the way the full audicle application is really amazing. Very visually interesting… World of warcraft style online live coding on the horizon. Share algorithms etc.

Very important synchronization code convention. You can synchronize applications by using this code in all running synchronized shreds. Uses modulo to quantize events:

.5::second => dur T;
T - (now % T) => now; //take now and use modulo

All threads have access to the same system clock To synchronize threads add the above code to the head of threads to be synchronized

Input devices

Use: Hid (HidIn deprecated) – get keyboard input

Chuck team has an interest in Commodity devices and using them for music making.

Chuck is open source and released under GPL. Note some of the waveguides may have patents associated with the algorithms. Patents expire shortly (they are from the early eighties). For the most part you are fine, you may want to check if you are distributing to millions of people.

Karplus Strong Implementation

Karplus and Strong Plucked string filter - Researchers at Stanford attempting to synthesize drum timbres - Julius Smith and David Jaffe(?) were inspired to look further - higher frequencies die off faster than lower frequency

//basic karplus strong algorithm
//although they used noise rather than an impulse
//feedforward
Impulse i => Delay d => dac;

//feedback with gentle lowpass
d => OneZero lowpass => d;

//generate the impulse
.5 => i.next;

//set the delay length
500::samp => d.delay; //this length controls the pitch

//dissipation
.99 => lowpass.gain;

//advance time
4::second => now;

You can currently package code such as the above into a chuck object. But currently you cannot currently make something like this into a unit generator.

look into the deep folder for interesting code examples:

THX emulater (amusing)
unclap.ck – Steve Reich clapping music realization

can talk to other applications using OSC bindings.

The Future

Another chuck workshop digesting advanced material is in the works for about two months from now through the Bay Area Computer Music Technology Group

Directed Graphs in Rails (Self Refering Many-To-Many Joins)

2007-10-22T17:22:00Z

Here is a simple example of a complex subject. An uncluttered implementation of a directed graph implemented in Rails (adaptable to other Ruby ActiveRecord projects).

Let the directed graph consist of nodes and links. Links join nodes together and are directed from a node to another node. Links also have a value associated with them.

I should mention that in standard digraph terminology nodes are called vertices, and edges are called links. I chose the terms nodes and links as I think it makes for easier reading.

Build The Model

Create the tables like this:

$ script/generate model node name:string
$ script/generate model link from_id:integer to_id:integer value:integer
$ rake db:migrate

The migration table definitions should look like this (surrounded by the usual migration code):

create_table :nodes do |t|
    t.column :name, :string
end

create_table :links do |t|
    t.column :from_id, :integer
    t.column :to_id, :integer
    t.column :value, :integer
end

Create the Node and Link models like this:

class Link < ActiveRecord::Base
    belongs_to :from_node, :foreign_key => "from_id", :class_name => "Node" 
    belongs_to :to_node, :foreign_key => "to_id",   :class_name => "Node" 
end

class Node < ActiveRecord::Base
    has_many :to_links, :foreign_key => 'from_id', :class_name => 'Link' #tricky!
    has_many :to_nodes, :through => :to_links  

    has_many :from_links, :foreign_key => 'to_id', :class_name => 'Link' #tricky!
    has_many :from_nodes, :through => :from_links
end

What’s Going On In The Model?

Did you see the lines labeled “tricky!” in the Node model. Why does has_many :to_links reference :foreign_key => “from_id”? Well, lets check it out.

Here is a diagram I created to explain directed relationships from one node to another node and how they are defined in the model. (Whispered aside: if you are feeling frisky and you want to compute the nodes that go to a given node rather than from it, you will have to traverse the diagram in reverse.)

Essentially we traverse in this order:

From the :from_node via the from_id to the :to_link
From the :to_link via the to_id to the :to_node

Here’s the corresponding code from the Node model that makes this hapen.

1) Traverse from the :from_node via the from_id to the :to_link

has_many :to_links, :foreign_key => 'from_id', :class_name => 'Link'

2) Traverse :to_link via the to_id to the :to_node

has_many :to_nodes, :through => :to_links

“Where are the :from_node and :to_node terms defined?” you may ask. They are defined in the Link model along with the belongs_to statements.

belongs_to :from_node, :foreign_key => "from_id", :class_name => "Node" 
belongs_to :to_node, :foreign_key => "to_id",   :class_name => "Node"

SQL Nitty Gritty

Lets disect what all of this means in terms of SQL. If you are ready to get right to it, you can skip this section and go straight to playing with the model.

In SQL selecting nodes linked from node “a” to node “b” would look something like the following. Notice that I alias the node table as both from_node and to_node (the node table plays two roles).

SELECT to_nodes.name 
FROM nodes from_nodes, nodes to_nodes, links
WHERE from_nodes.id = links.from_id
AND to_nodes.id = links.to_id
AND from_nodes.name = 'a'
AND to_nodes.name = 'b'

Looking at our diagram we see this bit of code from the Node model which defines the relationship from a Node to a Link:

has_many :to_links, :foreign_key => 'from_id', :class_name => 'Link'

This corresponds to this sql statment:

WHERE from_nodes.id = links.from_id

Looking at our diagram once again we see this bit of code from the Node model which defines the relationship between from a Link to a Node:

has_many :to_nodes, :through => :to_links

This corresponds to this sql statement:

AND to_nodes.id = links.to_id

One last note about the naming of the model. Although the symbols do represent an non-arbitrary relationship, the names for :from_nodes, :to_nodes, :from_links, and :to_links are arbitrary. I have selected a naming scheme that I believe reveals the structure of the relationships, but if you find something more suitable to your tastes ActiveRecord won’t complain.

Playing With The Model

Now that you have grokked that, lets try out the model. Here’s the diagram of the digraph that we are about to create:

Open up your rails app console:

$ script/console

Create some nodes and connect them with links:

a = Node.create(:name => "a")
b = Node.create(:name => "b")
c = Node.create(:name => "c")
d = Node.create(:name => "d")
Link.create(:from_node => a, :to_node => b, :value=>7)
Link.create(:from_node => b, :to_node => c, :value=>14)
Link.create(:from_node => c, :to_node => d, :value=>21)
Link.create(:from_node => b, :to_node => d, :value=>28)
Link.create(:from_node => d, :to_node => a, :value=>35)

Now run some queries.

a links to b:

a.to_nodes[0].name # returns: b
a.to_nodes.first.name #an alternate, also returns b

b links to c and d:

b.to_nodes[0].name #returns c
b.to_nodes[1].name #returns d
b.to_nodes.each {|node| puts node.name} # returns: c, d

There is a link to a from d:

a.from_nodes[0].name # returns: d

One way to retrieve a node two links away:

a.to_nodes[0].to_nodes[0].name #returns c

Get the value associated with a link between two nodes:

a.to_links[0].value #returns 7

There you go. Now you know how to make directed graphs in Ruby on Rails. Go forth and digraphify.

One final note, this posting was inspired by Josh Susser’s post on the same topic. He has some other great topics on “has_many :through” relationships – in fact that’s even what his blog is called. I recommend reading his posts if this type of model manipulation strikes your fancy.

CSS li Elements in Explorer

2007-09-27T01:59:00Z

Quick report. You may have a reason not to do this... but it may just bail you out. The vertical spacing of li elements can be quite wide on IE. This looked bad:

ul{
    padding-left:0;
    margin-left:0;
}

li{
    list-style-type: none;
    list-style-position: inside;
    marker-offset: 3px;
}

Adding this to the li tag fixed it: *vertical-align: bottom; width: 200px;* In context it looks like this:

li{
    list-style-type: none;
    list-style-position: inside;
    marker-offset: 3px;
    vertical-align: bottom;
    width: 200px;
}

Let me know if you run into side effects or if it works for you.

Bay Area Computer Music Technology Meetup: ChucK, Python VST, and More

2007-09-24T17:09:00Z

Bay Area Computer Music Technology Meetup: ChucK, Python VST, and More
Thursday, October 4, 2007, 6:00 PM
Space Gallery, 1141 Polk St , San Francisco, CA
Please RSVP

Come meet each other and participate in planned and impromptu presentations (“lightning talks”). Presentations will be Ge Wang and Spencer Salazar on ChucK, and Tim Thompson on his Python VST library. Bring your laptop if you would like to make a short impromptu presentation (projection screen and sound system available) or tell the group about new projects and opportunities.

PLANNED PRESENTATIONS

Ge Wang and Spencer Salazar presenting on the ChucK audio programming language. Ge and Spencer are both Alumni of the Princeton Sound Lab and Ge is joining the faculty at Stanford in the Center for Computer Research in Music and Acoustics (CCRMA). There is a lot of interest forming in the Computer Music community around ChucK and they have been at it’s originating point. Ge is the chief architect and co-creator of the language and Spencer is the author of several key components, as well as the miniAudicle, an IDE for enjoyable ChucKing. Ge is also enthusiastic about starting a West Coast Laptop orchestra.

Tim Thompson will be presenting on a library he has written that allows Python programmers to access the vast variety of VST plugins and synth instruments. He is a former longtime Bell Labs / AT&T / Network Appliance developer who has written a number of computer music frameworks (notably KeyKit). He has more recently been inspired by events such as Burning Man and Woodstockhausen, where his interactive creations include a 12-foot high lyre and an antique radio. Tim’s home page describes his projects and hosts his web-based algorithmic music toys.

Short “lightning talk” style presentations from users who have attended with their laptops. Contacting me in advance for this is not necessary unless you have specific concerns.

The event will be held at Space Gallery, 1141 Polk St in San Francisco. There will be a projection screen and stereo sound system. It is an enjoyable space and will be open for people to socialize after the presentations. There will be light food to enjoy while you are talking. Many thanks to Ray at Space Gallery for hosting this meetup!

At 9pm the venue will switch gears from Computer Music Technology Group tech talk. Further eclectic inspiration begins at 9pm as professional skateboarders do a short exhibition on a small portable half-pipe skate ramp inside the gallery while CB Records Mophone and friends perform mashup sets of live rock and funk re-mixes. Should be amazing!

Polk street also offers a variety of restaurants for after meetup dining and conversing.

Looking forward to meeting all of you, and please feel free to contact me about presenting for future events, hosting or sponsoring an event, etc. – noah [at] listenlabs.com.

BIOS

Ge Wang received his B.S. in 2000 in Computer Science from Duke University and PhD in 2007 studying with Perry Cook in Computer Science at Princeton University, and is an assistant professor at Stanford University in the Center for Computer Research in Music and Acoustics (CCRMA). Ge conducts research in real-time software systems for computer music, programming languages, visualization, new performance ensembles (e.g., laptop orchestras) and paradigms (e.g., live coding), interfaces for human-computer interaction, pedagogical methodologies at the intersection of computer science and computer music. Ge is the chief architect of the ChucK audio programming language and the Audicle environment. He is a founding developer and co-director of the Princeton Laptop Orchestra (PLOrk), and a co-creator of the TAPESTREA sound design environment. Ge composes and performs via various electro-acoustic and computer-mediated means.

Tim Thompson enjoys the creative process of developing artistic software for both music and visuals, often involving the use of unusual controllers. Most widely known as the developer of the KeyKit programming environment for algorithmic and realtime MIDI experimentation, he has more recently been inspired by events such as Burning Man and Woodstockhausen, where his interactive creations include a 12-foot high lyre and an antique radio. He lives in Silicon Valley and collaborates with local ensembles Dud and DOUBLE VISION. Tim’s home page documents the variety of his activities and allows people to play with web-based algorithmic music toys.

Spencer Salazar is a software engineer and music technologist from San Francisco. A cohort of the Princeton Soundlab, he is the main developer of miniAudicle, an integrated development environment for the ChucK audio programming language. He has implemented several components of ChucK proper, and has composed for, performed in, and developed software for the Princeton Laptop orchestra. By day he works on human-computer interface technologies for a Bay Area product development group.

ABOUT THE BAY AREA COMPUTER MUSIC TECHNOLOGY GROUP

Meet other computer music technology enthusiasts for presentations of computer music techniques and technologies (all music genre outputs are welcome). If you are a user of max/msp, csound, ChucK, CLM, Live, Reaktor, Kyma, JMSL, SuperCollider, or other computer music synthesis, composition, and live performance software this group is for you. Come to learn new techniques, better the code base, find collaborators, and find out what what’s happening in studios, labs, and live work spaces around the Bay Area.

Join the Bay Area Computer Music Group and RSVP for the first event here