Aquabu - Home

My Rubyconf 2009 Audio Talk Posted on Confreaks

2010-02-18T18:32:00Z

My talk at Rubyconf 2009 on using Ruby for audio projects was posted. I focus on frequently unfamiliar open source libs that people may find useful and make reference to a number of BArCMuT presenter's projects. There's a lot of talk about using Ruby and Chuck, SuperCollider, solving sync issues, etc, etc. You can find the source code for the presentations on "github here":http://github.com/aquabu <embed allowfullscreen="true" src="http://rubyconf2009.confreaks.com/player.swf" flashvars="image=images%2F20-nov-2009-17-15-making-music-with-ruby-patterns-context-fun-noah-thorp-preview.png&file=http%3A%2F%2Frubyconf2009.confreaks.com%2Fvideos%2F20-nov-2009-17-15-making-music-with-ruby-patterns-context-fun-noah-thorp-small.mp4&plugins=viral-1" allowscriptaccess="always" height="380" width="640" /> One major Ruby discovery since then is that Charles Nutter (a primary author of jRuby) has a version of Ruby that uses static typing, runs on the JVM, and runs at the speed of Java! It's called duby and it's available "here":http://github.com/headius/duby

BArCMuT Co-Organization Constitution

2010-01-24T22:28:00Z

Over the last year I have been contemplating how to further democratize the Bay Area Computer Music Technology group ( BArCMuT ) for more co-organization and stronger collaborative relationships, while maintaining a high quality of events. On my way back from NAMM this idea for writing a constitution for BArCMuT bubbled up. The only receptacle for the idea on South West was this slightly waxy emergency bag.

Since then I have refined the constitutions guidelines and listed them below. If you need to skim this article just skip to the “BArCMuT Constitution” section. Before I get to the guidelines, I would like to share some background on how I arrived at the new form for BArCMuT…

Some Background On BArCMuT’s Inspiration

In 2003 I attended a Dorkbot event in which Tim Thompson presented Key Kit game controller hacks and Dr. Friendly presented a crazy wavelet feedback algorithm. My friend Zu who is an entertainment magazine editor in Poland asked me what I thought the coolest thing in San Francisco was and I told her about Dorkbot. I meant it.

Then, in early 07 at the SF Ruby Meetup I had my mind blown by awesome technical talks by Chris Wanstrath (now Github co-founder) and others. These events made me wonder “why doesn’t a group like this exist specifically for the computer music community?”.

I had recently quit my job at Digidesign thinking that I was going to develop this weird ontological DJ remixing algorithm while I contracted. I was constantly looking for the right technology to build it in. My max patches had gotten un-managable with 100+ sub-patches and lots of logic that was more suited to procedural programming. Java audio sucked. Chuck – what was that? Csound was cumbersome. I was unsure about the Supercollider licensing. Ruby had no audio libraries. Python was a little better supported but less familiar. I wanted to hear people tell me about this stuff first hand.

I started the Bay Area Computer Music Technology Group in the fall of 2007. On the first night, the event magically came together as community leaders like Tim Thompson, Ge Wang, Vlad Spears, Scot Gresham Lancaster, and David Wessel decended on the Space Gallery in the Tenderloin of San Francisco. Nobody seemed to care about the street noise as we watched Ge’s chuck demo while sitting on church pews next to a recently assembled indoor skateboarding ramp.

The group took off after that night. Now there are over 500 members and we have had over 35 succesful events all over the Bay Area (Stanford CCRMA, Berkeley CNMAT, Mills College, Digidesign, Dolby, Maker Faire, GAAFTA etc). The presenters have been world class awesome and many of the members could just as easily be presenting or audiencing. I typically have to warn presenters that they will need to answer some very difficult questions from the expert tribunal refered to as the “audience”. Lots of people have told me how they acquired a job, met collaborators, changed product specs, or began an academic path at a BArCMuT event. This makes me happy and excited.

Fast forward to 2010. The community has gained a lot of great momentum with other groups like Phasor, Overlap.org, Learn Tech, Share San Jose, PyGame, and others. We’ve all been influencing eachother a lot and starting new projects. At an event at CCRMA, Tim Thompson said how BArCMuT had been one of his inspirations for starting the Share San Jose group. This was both nice and funny because I had actually been inspired by his talk at Dorkbot to create BArCMuT in the first place. I think this illustrates how a cool community can inspire itself.

Despite the fact that our community really is a headless web, BArCMuT has formed as a benevolent dictatorship. In 2010 I would really like that to change. To do that I am symbolically stepping down as benevolent dictator and setting up a structure for co-organizing the group (aka the constitution). In fact, it’s weird to even think of the group as just being “BArCMuT members”. BArCMuT is actually just a tool for the community to organize itself, and I would like to re-org BArCMuT to reflect this.

What’s In It For Me

Part of cooperation is understanding why the people you are cooperating with are doing what they do.

At the bottom of my South West emergency bag BArCMuT constitution, there’s a section titled “WIIFM” (an abbreviation for “What’s in it for me”?). At the risk of being overly explicit, let me share why I organize BArCMuT:

I want to participate in a community that I help to create
I want to meet cool collaborators and make cool s@&#%!
I want to know about awesome new electronic art technology, libraries, and techniques
I want you to think that I am cool and well organized and include me in cool projects
I want you to return my phone calls and emails

Conspicuously missing is any sort of profit motive. BArCMuT is an Open Source inspired organization, vis a vis Eric S. Raymond. The way I see it, collaboration creates value through network effects rather than through hording.

To get semi-technical, personal sub-optimization (greed) in a commons reduces the benefit of the commons as a whole and ultimately limits its abilities to benefit community members thus trashing the commons. For the mildly cynical or economics obsessed, building a commons can be thought of as “enlightened self interest”. But really, this technical framing doesn’t capture the spirit of why I built BArCMuT. For those who follow their heart, this approach can be described as just feeling good and creating community.

Envisioning The Computer Music Commons

To further the goals that I listed and I believe that many community members share, I want to reform BArCMuT as a durable commons for the computer music community. My strategy from the beginning has been to remove any frictions to having a neutral commons – such as such as pursuing direct monetary gain or gratuitous advertising. Now that BArCMuT exists as a community asset, the next step in reducing friction on the commons is to change the structure of BArCMuT from a tree to a graph.

Part of the strategy for limiting the politics that can exist within a group, is to limit the scope of what BArCMuT does. I hope to reduce politics by: not having a shared treasury, only allowing free events, and limiting the scope to events relevant to learning about or creating electronic arts (e.g. no concert audience promotion). BArCMuT definitely should support the excellent commercial work that is being done within our community ecosystem, but as a commons it should be clearly differentiated from these ventures.

The BArCMuT Constitution

Every group needs some guidelines – the more lightweight the better. Here’s my first attempt at codifying a group constitution for a co-organized BArCMuT. Please post your feedback.

Goals of the constitution:

Create a durable commons that is not reliant on a single member (i.e. not a benevolent dictatorship run by me)
Further enable the Bay Area computer music community to stay informed about technical talks, hack sessions, and community events
Create a neutral commons for members to find collaborators, employment, teachers, and students
Filter events to maintain a high level of quality

Participants:

There are four general participant types: co-organizers, hosts, presenters, and attendees
Initial co-organizers will be invited by me
Additional co-organizers can be added or removed by an 80% group vote (I could use some additional advice on the official decision making process).
Co-organizers can schedule official BArCMuT events if they fit the guidelines. An official BArCMuT event is one that sends RSVPs and is listed on the main page of barcmut.org
Co-organizers, hosts, and presenters are encouraged to list their logo on the barcmut home page

BArCMuT event guidelines:

events must include a technical talk about electronic art technology
events must be free to members
events must be in the Bay Area
hosts, presenters, and co-organizers are encouraged to mention upcoming electronic art events (free or not), job positions, and products. Although community relevant promoting is encouraged as a vital part of the ecosystem, this should not dominate the event.
co-organizers can list events (even if they are not official BArCMuT events) in the BArCMuT calendar. It would be great if we could announce all upcoming events listed in the calendar at official BArCMuT events.
there will be no shared treasury or treasurer for BArCMuT
venues, presentations, and co-organization are offered for free
expenses will be payed by co-organizers or by donation (e.g. for security gaurds, snacks, etc.)

I am inviting trusted community members who I have organized events with in the past to be BArCMuT co-organizers. These co-organizers can schedule events that fit the BArCMuT guidelines. These include:

Ge Wang (CCRMA, Smule)
Sasha Leitman, Linnea Williams, Carr Wilkerson (CCRMA)
Tim Thompson (Share San Jose, Visual Music Meetup, ZeroOne)
David Wessel, Adrian Freed, Andy Schmeder (CNMAT)
Barry Threw (GAFFTA, Overlap.org, etc.)
Vlad Spears (Phasor, Overlap.org)
Harry Tormey (PyGame, Digidesign)
Chris Brown (Mills)
Scot Gresham-Lancaster (Cogswell, The Hub)
Rich & Moldover (Learntech, LoveTech)
If you are not on the list and you think you should be let me know

A note about scheduling. If there are too many events (e.g > 4 per month) we may need to thin things out. One strategy would be to to list events that repeat frequently once as official BArCMuT RSVP events and then list them in the non RSVP calendar to be announced at other events.

Conclusion

In conclusion, I am excited about being part of more great events this year as a co-organizer. My plan is to work towards a durable commons through transparency and establishing light weight and usable guidelines. If I am doing something that seems to work against this plan, please tell me. I’m not perfect and I would appreciate the feedback.

Even before this article has been completed I have had some great conversations with other organizers about creating this commons. 2010 is going to be a great year for community!

jQuery Wherefore Doth Thy Includes Go?

2009-07-21T03:15:00Z

h2. Whither Javascript Recently I have been working on a Rails project with lots of Javascript include complexity ( "Ashbury Music Hall":http://www.ashburymusichall.com ). Here's the challenge: * Some larger libraries and plugins only need to appear on single pages (e.g. Thickbox, FCKEditor). * Some Ajax page renders depend on core libraries (e.g. jQuery) but shouldn't be loaded more than once by layouts and partials * When Ajax page renders degrade to linked pages they may be in alternate layouts and include different Javascript libraries or have different load orders * Rails partials load *before* layouts and potentially include Javascript files specified in them before core libraries * Page and partial specific Javascript (e.g. myview/my_page.js ) presents even more complexity * Pages appear to load faster if Javascript is loaded on the bottom of the page Oh what a tangled web we weave. h2. Thither The solution I arrived at is to specify sectional order of libraries, accumulate them, deduplicate them, and include them at the end of the page request. This allows definition of core libraries at the top of a layout page for all partials and inclusion of librararies in plugins on any view or partial without duplicate loading or core library omissions. To do this I defined two Application Helpers as follows:

# Use accumulate_js to accumulate javascript paths throughout the application
# it deduplicates them and then they should be included by calling include_accumulated_js
# at the foot of the page.
def accumulate_js(key, path)
  @javascript_list ||= {} 
  @javascript_list[key] ||= []      

  if path.class == Array || path.class == String
    @javascript_list[key].<<(path)
    @javascript_list[key].flatten! # for some reason + was not working properly for arrays
    @javascript_list[key].uniq!
  else 
    raise ArgumentError
  end
  @javascript_list
end

# call accumulate_js first to use the default @javscript_list instance var
def include_accumulated_js(order_array, path_hash = @javascript_list)
  result = ""

  order_array.each do |key|
    next unless path_hash[key]
    result += path_hash[key].inject("") do |result, path|
      result += javascript_include_tag(path)
    end
  end
  result
end

Then wherever Javascript is called for a request (layout, view, or partial) the names of the libraries are accumulated:

accumulate_js(:core, ["jquery-1.3.2.min.js", "jquery-ui-1.7.1.custom.min.js"])
accumulate_js(:plugins, "beautytips/jquery.bt.min.js")
accumulate_js(:application, "application")

All load path strings from the public/javascripts directories are specified. The library order is maintained but additional loads are de-duplicated. Then all of the libraries are loaded at the foot of the document, ordered by type with a command like this:

include_accumulated_js([:core,:plugins,:application,:page,:partial])

For the specifics of what it does, here's the rspec spec:

describe "javascript library acuumulation" do
  it "accumulates javascripts associated with a specific key" do
    accumulate_js(:core,["a","b"])
    accumulate_js(:core,"c").should == {:core => ["a","b","c"]}
  end

  it "can accumulate a list of javascripts used on a web page" do
    accumulate_js(:core, "library.js")
    accumulate_js(:core, "another/library.js").should == {:core => ["library.js", "another/library.js"]}
  end

  it "can take combinations of arrays and strings" do
    accumulate_js(:foo, ["lib1.js","lib2.js"])
    accumulate_js(:foo, "lib3.js")
    accumulate_js(:foo, ["lib4.js","lib5.js"]).should == {:foo => ["lib1.js","lib2.js","lib3.js","lib4.js","lib5.js"]}
  end

  it "raises an error if it is of an unknown type" do
    lambda do
     accumulate_js(:foo, 1)
    end.should raise_error(ArgumentError)
  end

  it "deduplicates and preserves order" do
    accumulate_js(:baz, ["a","b","a","c","b","c"]).should == {:baz, ["a","b","c"]}
  end
end

describe "including accumulated javascript" do
  before(:each) do
    accumulate_js(:foo, ["foo_1","foo_2"])
    accumulated = accumulate_js(:baz, ["baz_1","baz_2"])
    @result = include_accumulated_js([:baz, :foo], accumulated)
  end

  it "returns javascript includes" do
    @result.should =~ /script.*src.*javascripts\/baz_1.js.*type.*text\/javascript/ 
  end

  it "includes javascripts in the specified order" do
    @result.should =~ /baz_1.*baz_2.*foo_1.*foo_2/m
  end
  it "can handle nil values" do
    accumulated = accumulate_js(:foo, ["foo_1","foo_2"])
    lambda do
      include_accumulated_js([:baz, :not_in_there], accumulated)
    end.should_not raise_error
  end
end

h2. Accolades "Marc Grabanski":http://marcgrabanski.com, excellent jQuery developer that he is, contributed his sage advice to this solution. h2. Epilogue I might put this in a plugin and post to github, but just seems a little silly since there are only two helper methods. What think ye? If this blog post is good enough for you, then it's good enough for me. Obviously, I'm busy enough that I've gone bonky and resorted to archaic english - but if you request a plugin or some other container I may be obliged.

Ruby Value To Boolean Method

2009-06-25T23:40:00Z

It's a common problem that does not seem to have a standard solution. Sometimes you need to coerce a value to a boolean (e.g. from a web form). Here's a simple implementation I wrote that handles common strings and numbers and gives flexibility in how to treat nils. Here's the behavior we want:

  describe "#to_boolean" do                                                                         
    it "converts false values" do                                                                   
      ["no","false",false, "0", 0].each do |value|                                                  
        to_boolean(value).should == false                                                           
      end                                                                                           
    end                                                                                             
                                                                                                    
    it "converts true values" do                                                                    
      ["yes","true",true, "1", 1].each do |value|                                                   
        to_boolean(value).should == true                                                            
      end                                                                                           
    end                                                                                             
                                                                                                    
    it "is case insensitive for strings" do                                                         
      to_boolean("YeS").should == true                                                              
      to_boolean("NO").should == false                                                              
      to_boolean("TruE").should == true                                                             
      to_boolean("FalSe").should == false                                                           
    end                                                                                             
                                                                                                    
    it "converts nil to false by default" do                                                        
      to_boolean(nil).should == false                                                               
    end                                                                                             
                                                                                                    
    it "can return nil as nil" do                                                                   
      to_boolean(nil, nil).should == nil                                                            
    end                                                                                             
                                                                                                    
    it "converts unmatched strings to true" do                                                      
      to_boolean("a string").should == true                                                         
    end                                                                                             
  end

Here's the source code that satisfies the desired behavior:

def to_boolean(value, nil_value = false)
  value.downcase! if value.class == String
  case value
  when "no","false",false, "0", 0
    false
  when "yes","true",true, "1", 1
    true
  when nil
    nil_value 
  else
    !!value
  end
end

restful_authentication and rspec-rails 1.2.2

2009-03-26T01:36:00Z

If you are getting something like this:

Test::Unit::AssertionFailedError in 'SessionsController route generation should route the create sessions correctly'
The recognized options <{"action"=>"show", "controller"=>"sessions"}> did not match <{"action"=>"create", "controller"=>"sessions"}>, difference: <{"action"=>"create"}>
./spec/controllers/sessions_controller_spec.rb:108:

You need to fix your restful_authentication specs to something like this (from my git diff):

     it "should map #edit" do
-      route_for(:controller => "assets", :action => "edit", :id => 1).should == "/assets/1/edit" 
+      route_for(:controller => "assets", :action => "edit", :id => "1").should == "/assets/1/edit" 
     end 

     it "should map #update" do
-      route_for(:controller => "assets", :action => "update", :id => 1).should == "/assets/1" 
+      route_for(:controller => "assets", :action => "update", :id => "1").should == {:path => "/assets/1", :method => :put}
     end

Essentially you need to change your == match to be a hash which includes a :method key. Also, numbers for the :id param should be quoted.

This will probably be fixed on the restful_authentication side since David Chelimsky has posted a patch on github (thanks David!). But, if you created these specs with the restful_authentication generator you may want to just go in and change them yourself. Onward!

My Band Rabbit's Rum is an Independent Music Awards Finalist

2008-11-20T07:22:00Z

My music project "Rabbit's Rum":http://www.rabbitsrum.com has just been nominated as an Independent Music Awards finalist. "Check it out!":http://tinyurl.com/6mtzfh

ActiveRDF and Redland - Up and Running

2008-11-04T19:04:00Z

Conveniently the Redland libraries have been added to macports. To install them you will need to install both the Redland and Redland-Bindings library. In my case I installed them with the +ruby option although it's hard to tell if this is required given sparse documentation:

sudo port install redland +ruby
sudo port install redland-bindngs +ruby

Then to test that Redland is working it is recommended that you run the example.rb file. Since it's good to have the source around anyway you can get it from the Redland subversion repository here:

svn export http://svn.librdf.org/repository/bindings/trunk/

To run the example and make sure things are working, cd into the ruby directory and enter this per the "Redland Ruby Binding instructions":http://librdf.org/docs/ruby.html :

ruby -I. -Ilib example.rb file:../data/dc.rdf rdfxml

You should get output like this:

found statement: {[http://www.dajobe.org/], [http://purl.org/dc/elements/1.1/description], "The generic home page of Dave Beckett."}
found statement: {[http://www.dajobe.org/], [http://purl.org/dc/elements/1.1/creator], "Dave Beckett"}
found statement: {[http://www.dajobe.org/], [http://purl.org/dc/elements/1.1/title], "Dave Beckett's Home Page"}
Parsing added 3 statements
Printing all statements
Statement: {[http://www.dajobe.org/], [http://purl.org/dc/elements/1.1/title], "Dave Beckett's Home Page"}
Statement: {[http://www.dajobe.org/], [http://purl.org/dc/elements/1.1/creator], "Dave Beckett"}
Statement: {[http://www.dajobe.org/], [http://purl.org/dc/elements/1.1/description], "The generic home page of Dave Beckett."}
Querying for dc:titles:
{
  0= [http://www.dajobe.org/]
  1= Dave Beckett's Home Page
}
Serialized query results to JSON as a string size 296 bytes
Writing model to test-out.rdf as rdf/xml
Serialized to ntriples as a string size 292 bytes
Done

Also you should have the test output in the test-out.rdf file. Something like this:

<?xml version="1.0" encoding="utf-8"?>
<rdf:rdf xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <rdf:description rdf:about="http://www.dajobe.org/">
    <dc:title>Dave Beckett's Home Page</dc:title>
  </rdf:description>
  <rdf:description rdf:about="http://www.dajobe.org/">
    <dc:creator>Dave Beckett</dc:creator>
  </rdf:description>
  <rdf:description rdf:about="http://www.dajobe.org/">
    <dc:description>The generic home page of Dave Beckett.</dc:description>
  </rdf:description>
</rdf:rdf>

Cool! Next, lets setup ActiveRDF. If something goes wrong in your install or you want more information, the ActiveRDF install instructions are "here":http://wiki.activerdf.org/GettingStartedGuide Now, for ActiveRDF install these gems:

sudo gem install activerdf --include-dependencies -y
sudo gem install activerdf_sparql
sudo gem install activerdf_redland

Don't forget the activerdf_redland gem or you will get the error: unknown adapter redland Enter this code from the ActiveRDF getting started "guide":http://wiki.activerdf.org/GettingStartedGuide into a file called *active_rdf_demo.rb*:

require 'rubygems'
require 'active_rdf'
adapter = ConnectionPool.add_data_source :type => :redland
adapter.load "test_person_data.nt"
Namespace.register :test, 'http://activerdf.org/test/'
ObjectManager.construct_classes

# we can access all RDF properties of a person as Ruby attributes:
 eyal = TEST::Person.new 'http://activerdf.org/test/eyal'
 puts eyal.age
 puts eyal.eye
 puts eyal.class

 # lets create an instance of one of the classes, that were constructed in that 
 # way armin will be of rdfs:type test:person
 armin = TEST::Person.new 'http://armin-haller.com/#me'

 # now lets search for something in the triple store
 all_resources = RDFS::Resource.find_all

 # print all the people, and their friends
 all_people = TEST::Person.find_all
 all_people.each do |person|
     puts "#{person} has #{person.eye} eyes" 
 end

 # find all people aged 27
 almost_thirties = TEST::Person.find_by_age(27)

 puts "the following people are almost thirty: #{almost_thirties}"

Download the test data "here":http://activerdf.org/data/test_person_data.nt and put it in the same folder as your active_rdf_demo.rb file. Run active_rdf_demo.rb:

ruby active_rdf_demo.rb

You should get some output like this:

27
blue
TEST::Person
<http: /> has blue eyes
the following people are almost thirty:

Congrats! ActiveRDF is now working with Redland using the Redland Ruby bindings. Now if by chance you get this error next time you run script/server for a rails app:

/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require': no such file to load -- /opt/local/lib/ruby/gems/1.8/gems/activerdf-1.6.10/lib/activerdf/init.rb (MissingSourceFile)

This is because some versions of the ActiveRDF gem (e.g. 1.6.10) do not have an init.rb file created. This is a bug. Here's how you can get around it (the directory to cd into will be the one from your error):

sudo su
cd /opt/local/lib/ruby/gems/1.8/gems/activerdf-1.6.10/lib/
mkdir activerdf
touch activerdf/init.rb

Scruffy Ruby Gem Debugging

2008-07-16T09:25:00Z

!http://www.aquabu.com/assets/2008/7/16/cast.jpg! In the process of creating some graphs in ruby I decided to go upstream from ruport and use the scruffy gem directly. I made some discoveries along the way that touch on issues that could be encountered by various permutations of scruffy, macport, and rubygems users. h2. A Storm Brewing The first issue I encountered was this error:

testing_scruffy.rb:2:Warning: require_gem is obsolete.  Use gem instead.
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems.rb:304:in `report_activate_error': Could not find RubyGem Scruffy (>= 0.0.0) (Gem::LoadError)
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems.rb:238:in `activate'
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems.rb:76:in `active_gem_with_options'
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems.rb:61:in `require_gem'
	from testing_scruffy.rb:2

This seems like a straightforward case of not installing the gem. However, in my case the the gem *was* installed and there was additional strange behavior: require 'scruffy' would load the library in irb but not from a script (even with require 'rubygems' before it).This seemed inconsistent because $LOAD_PATH (aka $:) variables were identical in irb and in the script. After investigating a number of possibilities I updated rubygems (which was at 0.9.4) to 1.2.0 and this seem to resolve the blatantly bizarre behavior. You can check which version of ruby gems you are running with "gem -v". If you are using macports you may have an older version of ruby gems, in which case you may find it useful to update rb-rubygems. Caveat emptor: dependent updates may also be installed - such as ruby. Here's how you can update your macport rb-rubygems library on OS X:

sudo port upgrade rb-rubygems

h2. Scruffy Needs A Shave If you do upgrade ruby gems you may run into the another issue. With an updated version of ruby gems the require_gem method is obsolete. Because of this you may see the following error since scruffy 0.2.2 is not updated to use the *gem* method instead of *require_gem*:

NoMethodError: undefined method `require_gem' for main:Object
	from /opt/local/lib/ruby/gems/1.8/gems/scruffy-0.2.2/lib/scruffy.rb:18
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:32:in `gem_original_require'
	from /opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:32:in `require'
	from (irb):3

I used the hack below to get around this and to avoid unpacking the gem in to my project directory for modifications. Once again caveat emptor - you may ultimately be better off unpacking (freezing) the gem into a local directory and including it in your application directly. Here's the hack which simply creates an alias to suspend the deprecation of the require_gem method indefinitely:

require 'rubygems'
alias require_gem gem #later versions of gem use gem instead of require_gem
require 'scruffy'

h2. Case Of The Missing Pie Ah, but that's not all. It appears that tutorials and documentation are out of sync with the rubygem download... see example code at "http://scruffy.rubyforge.org/":http://scruffy.rubyforge.org/ which suggests this usage (with my require hack at the top):

require 'rubygems'
alias require_gem gem
require 'scruffy'

graph = Scruffy::Graph.new
graph.title = "Favourite Snacks"
graph.renderer = Scruffy::Renderers::Pie.new

graph.add :pie, '', {
      'Apple' => 20,
      'Banana' => 100,
      'Orange' => 70,
      'Taco' => 30
}

graph.render :to => "pie_test.svg"
graph.render :width => 300, :height => 200, :to => "pie_test.png", :as => 'png'

Running the sample code throws this error:

test_scruffy.rb:7: uninitialized constant Scruffy::Renderers::Pie (NameError)

Well, this error occurs because there is no such class as Scruffy::Renderers::Pie. It's not in the docs and if you do grep -r "pie" . from within the scruffy-0.2.2 folder it comes up with nothing that would lead to the creation of such a class. FYI, the docs are visible online here "http://scruffy.rubyforge.org/doc/":http://scruffy.rubyforge.org/doc/ Well, it's hard to use a class that doesn't exist. It appears that this may be present in scruffy 0.2.3 released on July 13th - but as of this writing it's not downloaded via sudo gem update scruffy. If you need this and it it is not yet available through ruby gems - you may need to do a manual install. In the mean time let's try something else instead. h2. Steady Into Port (Mind The Wireshark)

graph = Scruffy::Graph.new
graph.title = "Sample Line Graph"
graph.renderer = Scruffy::Renderers::Standard.new

graph.add :line, 'Example', [20, 100, 70, 30, 106]

graph.render :to => "line_test.svg"
graph.render  :width => 300, :height => 200, :to => "line_test.png", :as => 'png'

Well if you have RMagick installed that may have worked for you. If not you will need to make sure rmagick is installed. On my machine I had to remove some dependencies before reinstalling tiff with different compilation options. Here are the commands I ran. You will probably have different dependencies, so know what you are doing. Don't just blindly run these commands or you may uninstall something you need:

sudo port uninstall wireshark
sudo port uninstall gtk2
sudo port uninstall tiff @3.8.2_1+macosx
sudo port install tiff -macosx imagemagick +q8 +gs +wmf
sudo gem install rmagick

Note that installing and compiling rmagick took several minutes. I would suggest looking at the "rmagick installation page":http://rmagick.rubyforge.org/install-osx.html if you need to install this. At this point you should be able to output your svg and png files. h2. Conclusion Careful, or due to "environmental conditions" you could set out expecting a simple three hour tour happily outputting beautifully rendered svgs as you sail effortlessly through graph land; but instead, end up whistling along sympathetically to the Gilligan's Island theme song as you set ground on the shore of an uncharted desert isle. Hopefully this will help you through some of the things you may run into, and you can have a smooth sail enjoying scruffy.

Proc.new, lambda, and proc

2008-06-18T17:42:00Z

In response to the "harmony of thought posting on good ruby interview questions":http://blog.ritirisi.com/2008/06/17/15-questions-to-ask-during-a-ruby-interview I wrote the following (posted here for interest and to preserve formatting)... One thing about lambdas and procs. Even though they appear to be the same class they do not behave completely the same. From running my_lambda.inspect and seeing the Proc instance you would never guess it but they are different. Essentially lambda and Proc.new handle arguments and the "return" statement differently. Also the "proc" method returns a lambda not a Proc. Here's some code to prove the point. The argument handling is the easiest to demo:

my_proc = Proc.new {|x,y| print x,y}
my_lambda = lambda {|x,y| print x,y}
my_other_proc = proc {|x,y| print x,y}

# pass in one variable instead of 2
my_proc.call(1) #this doesn't throw an error and sets y to nil
my_lambda.call(1) #this throws an error
my_other_proc.call(1) #this throws an error like a lambda because it actually is a lambda not a proc in ruby 1.8.6

Also in Ruby 1.9 I think that the "proc" method (as seen in line three) will actually return a Proc and not a lambda. Ruby 1.9 will also add a "lambda?" method for you to clarify the difference between a Proc and a lambda. O'Reilly's book "The Ruby Programming Language" by Matz (Yukihiro Matsumoto) and David Flanagen has a great chapter on this.

Railsconf 2008 Scribblings: Hosting and the Woes (Engine Yard Panel)

2008-06-13T23:43:00Z

5/30/2008, 11:45AM, Portland, OR
Here are some of my notes from Railsconf 2008. Warning these notes are only vaguely edited (but arguably still useful). There are certainly typos and errors. If executing code from one of these articles, your mileage may vary (from spontaneous self destruction of everything you know and love to spontaneous coolness). Also, you may want to double check â€œfactsâ€. You definitely should not use these articles as a spelling reference. Feel free to post corrections. Engine Yard Panel: * Tom Mornini - CTO * Taylor Weibley - Application Support Director * Edward Muller - Automation Manager (fluent in binary) * Ezra Zygomowitz - System Architect * Jamie van Dyke Problems they have had and how they solved most of them. Then questions and answers. h2. Main problem with rails is active_record Great for quickly building with RESTful resources. But... * find(:all).each do |leak| * testing is usually with 20 records. * no indexes. Engineyard has a team of SQL guys to optimize in this area. One of the first problem they noticed that you need indexes on all foreign keys. If you don't have them they are full table scans. Migrations automatically index ids but not foreign keys. Basically anything that ends with _id needs an index. There may be some database experts that will disagree but this is not usually the case. Use explain in mysql /G to make sure that the indexes are being used in the queries. Take a look at the new Relic optimization stuff it's super cool. h2. Plugins that give common problems * Most common and hated plugin is ferret. Indexes grow past 1G and indexes frequently corrupted. Use sphinx instead. Sphinx will just update indexes on the delta. * Image Science uses ruby inline which needs environment variables to be set. Can be used perfectly in development and then when they shift to production monit clears those variables out. Not a huge problem but you should be aware of it. * Hodel 3000 loggers default is a verbose logging tool and it can generate huge log files. Adjust settings. Logging should be set to info. h2. How much traffic can you take? *Depends on coding* General stats: * Digg 10,0000+ visitors and 1% signup. What should you do? Cache it. At engine yard a slice could handle this quite easily. Customers are often concerned with this sudden out of the blue Digg or TechCrunch. But in reality you usually are and there is no surprise. Makes it so you can be ahead of the curve. * TechCrunch 1000+ visitors but they are more interested in what you are doing. * Ruby Inside / Flow 500 - 1000 visitors a day, also don't get many sign ups. It's more about what can the visitor do. * The Today Show - 100,000+ visitors in the first hour or so. Can get up to 10,000 signups in the first couple hours. You will have advance warning which is great. * Fox Business - 2000+ connections. h2. How many queries can your site take? It depends on code. * Queries per page * But you should cache! * Best way to handle hike in traffic is cache / memcache * Keep file input output down as low as possible * Separate out to different disks, you don't want simultaneous reads and writes h2. Deployment Lots of customers not up to speed on Capistrano, git / subversion. Engine yard gives deploy.rb (shameless plug from them :-) gem source - a http://www.engineyard.com ... Includes cap examples list of extra tasks - loads of things that it can do. h2. ebb | mongrel | thin. Which do customers prefer? Alternatives to mongrel are possible but not necessarily production ready (ebb). Don't need to be super concerned about it unless you are doing 1,000 hits per second. The bottleneck is usually the user code, this is far more relevent. That said, thin may be 13% faster but you need to be in the top 5% of internet traffic for this to matter. h2. Audience Q&A What happened with NGINX? You haven't heard about much new because it's in use and it just works. They haven't really had any problems with NGINX. Some minor issues but they have been ironed very quickly. They have seen 40MB/sec of static images served and not show up top command. Interpreters?
Rubinius and Modrubinius are going to be the way to go in the future. Thin instead of mongrel because of lower memory footprint?
There are a lot reasons to save memory but mongrel is a minor concern. More than 3-4 mongrels per CPU is pointless. The bottleneck is file io rather than memory. A common misconception is that you need more mongrels. Thin tends to backup processes more than mongrel. It is important to watch the real behavior of the app because every app uses resources differently. Should static files be local or shared across cluster?
Cluster is fine @ Engine Yard, but depends on file system. NFS was cool in 1979... How to handle asynchronous background processes?
Background Job (BJ) is pleasurable to work with. Written by NGINX. Do you leave keep alive with NGINX?
Untested. But historical use with apache pre-rails was not succesful. Seperate server for search workload? How should resources be distributed across server farm and how does this map to virtualization? Engine Yard are strong believers of virtualization. Intended from earlier on to only have virtualized servers. The more that they have gotten involved in it they have discovered that if you have a well built virtualization environment you quickly get the sense that you are operating on separate machines. NGINX, 3 Mongrels, and an memcache per instance. If you have a solid architecture you could have databases on multiple instances. Single server rack at engine yard with 18 servers would run bottom 97% of all internet apps. There's huge CPU capacity in a virtualization environment. CPUs are not typically the bottleneck (unless you are doing fluid dynamic calculations). What is the Engine Yard uptime?
3-4 9s up time. Rack Space rates using a questionable rating system.

Railsconf 2008 Scribblings - MagLev: Ruby That Scales

2008-06-13T23:26:00Z

Talk with Avi Bryant and Bob Walker
5/30/2008, 3:40PM, Portland, OR
Here are some of my notes from Railsconf 2008. Warning these notes are only vaguely edited (but arguably still useful). There are certainly typos and errors. If executing code from one of these articles, your mileage may vary (from spontaneous self destruction of everything you know and love to spontaneous coolness). Also, you may want to double check "facts". You definitely should not use these articles as a spelling reference. Feel free to post corrections. MagLev is rails that go fast. Typically have mongrel cluster talking to storage device and memcache instance for each server. Maglev unifies this stack: Storage, Cache, and Ruby Example: "Ordinary ruby shells". Two terminals using the same shared objects! You can do this for any number of boxes across the entire system. Local state is not shared, global variables ($hat). Ruby shell has transactions as you are used them. Transactions are there for mutable state.Commits don't only go to in memory caching, they go to disk and if there is a failure then the machines can be rebooted with the existing state. Could this be on multiple machines? Yes. The storage can be greater than ram because they can be persisted to disk. The system does not currently run rails, but it will be soon. Can currently run webrick with hello world. Question: Is the ruby spec implemented and at what level? Next phase is implementing the ruby specs against this. Question: What happens to files and sockets? Each VM will have it's own seperate copy of that resource and they will be nil'ed out when brought over. Question: Is this a written from scratch ruby machine? Yes, it is not MRI. Gemstone founded in 1982 in Portland. * Cretated GemStone smalltalk * Enterprise proven technology * Production systems deployed worldwide * Support domains that are not well served by relational database systems * Worldwide container * Intelligence * Telecom * Etc. MagLev * 1 dual core up to really large systems * Mutliple ruby virtual machines * Oject Caches/Repository * Transactional * Distributed * Scalable They are currently 100 days into the implementation process. * Dynamic Language VM * Based on Gemsone/s 64VM * Well tuned garbage collection * Extended with new byte codes for Ruby * JIT native code generation Object Caching * Shared caching uses shared memory * Repository objects are read into the cache ... Transactional * Fully recoverable ACID Transactions * Simple API * Conflict management Persistence * Transparent * Native Language * Simple API * You can store a trillion instances * Up to 17PB Benchmarks * bm_so_exceptions 5.6 x faster than MRI * fib 12.5 x MRI * vm2 method 24 x MRI * bm_so_sieve 68.4 times * bm_loop_whileloop2 111.5 x MRI "Holy Shit" -Antonio Gangiano (Great Ruby Shootout) Numbers will be independently verified shortly. Audience question. What is missing from Ruby that will decrease this performance? The plan is ruby spec and core library. But does it scale? Yes. What's the vision? Rails - taking it to a different level of scalability. What Else? * Open Community Involvement * Ruby Spec Compliance * MagLev Ruby open source * Free version of MagLev will be feature limited for small to medium scale * Info @ maglev.gemstone.come A lot of rails makes use of the relational database structure and indices. Will Maglev address these? Yes. And you will be able to make your queries in the native language. Will likely clone the API but run gemstone underneath. Continuations? Yes. They are supported in smalltalk. How to optomize? Will it be a linear scan? Gemstone does will do an linear scan for a basic select but gemstone does have various indexing. Will the Gemstone VM be open sourced (asked by Charles Nutter from jRuby team): Bob: "In a world of infinite possibilities it's possible." Will existing extensions in C be possible to integrate? Not currently. Rubinius has done some good work in this area and it may be possible to use a similar approach for Gemstone. C extensions may not be necisary in certain cases. However rails depends on C like BigDecimal(Wilson)? Will solve those problems when they get to them. ActiveRecord uses SQL fragments. How would that work since it's not part of ruby? To port a rails app may require getting rid of optomizations done in sql.... this is an area where feedback will be welcome. This is yet to be designed built. Which pieces are open source? All of the stuff at the ruby level is hopefully to be shared with Rubinius. ActiveRecord API will be. But as for the underlying VM Bob answered that earlier (e.g. "in a world of infinite possiblities...."). Is the persistent store distributable? More like typical SQL statement. Master slave replication... How is the VM implemented? In short... in C. When is going to be ready? Pay attention to what we are doing. More info in September.

Railsconf 2008 Scribblings: Joel Spolsky (Keynote)

2008-06-13T23:17:00Z

5/30/2008, 9:15am, Portland, OR Here are some of my notes from Railsconf 2008. Warning these notes are only vaguely edited (but arguably still useful). There are certainly typos and errors. If executing code from one of these articles, your mileage may vary (from spontaneous self destruction of everything you know and love to spontaneous coolness). Also, you may want to double check "facts". You definitely should not use these articles as a spelling reference. Feel free to post corrections. Intro by DHH on Joel: * Joel is author of Joel On Software * Generated programming language to generate other programming language * fighting complexity (anti-architect astronaut) h2. Joel's Talk Begins... Why does Brad Pitt make so much more money and have a bigger brand than Ian Somerhalder? What becomes "blue chip software" rather than off brand? Herman Miller Aeron Chair was the bluechip chair. All types of people make clones. Clones didn't have market share. This happens in every field. Happens in music as well. Great software has three components that make them blue chip: * Make people happy * Obsess over aesthetics * Observing the culture code h2. Make People Happy (Agency) [Very funny example about logging in to windows to post a photo and then having to install windows downloads and then use original install a file from the insert original install disk. In a word you feel enslaved (you are not in control of the process)] Martin Seligman pioneering psychologist of "Learned Helplessness" liked to study happy people which is rare amongst psychologists. Example: Someone dies > you are stuck in bed depressed > you loose your job and that you begin to believe that you lack agency and then you begin to teach yourself that you are helpless. Seligman's strategy was to teach people that they have power over there environment through tricking the brain into a virtuous cycle of having agency (initially through accomplishment of simple tasks). Abercrombie ecommerce checkout forces you through 4 pages. Amazon gives you all the options at once (change address, shipping options, etc). This is Joel's first example of giving the user agency. Make people happy by putting them in control (speed of response is also important here - the business case for AJAX). h2. Obsess over aesthetics Samsumg Blackjack vs iPhone. * Blackjack faster * iphone bricks if you install third party software * Blackjack ugly & iphone is seamless and shiny (if you accidentally swallowed one it would go right down) * Blackjack replaceable battery - iphone doesn't (avoids the non aesthetic seam). Another example is looking at the bottom of the MacBook vs. thinkpad. Apple design decisions are about fashion. Another example is a historical Paris apartment building in Paris has no fire escapes. It turns out fire escapes are zoned, but the appartment buildings are reclasified as monuments and they are not inspected. This is another example of taking aesthetics as more important than core functionality. Programers call this lipstick and say I want to work in the guts (the 90% of the iceberg below the surface). But it turns out that aesthetics are very important. To programmers it is not clear why artists such as Basquaiat etc. sell paintings for $50+ million. Specifically programmers don't get modernist architecture. Modernist principles remove all decoration from architecture. There is a parallel here between the programers gravitation to the command line vs. glossy operating systems. h2. The Culture Code Example: the promotion of sport utility vehicles. Deaths per million cars: * Toyota Camry 41 * Ford Explorer 88 This has to do with a number of factors. But you feel safe in these vehicals. This has to do with promotion (see the culture code). * Everything around you should be round and soft * You should be up high (on the reptilion level if you feel tall you feel you are dominating) * Key element of safety as a child is being fed if everything is round, soft, high, and you feel fed (cupholders) you feel safe. Another example like this is the word "Enterprise". Recalls tokyo at night, a clean server room, the starship enterprise, people in clean business suits looking up(image). Another example is Web 2.0. Lots of logos. They don't have visions they go to parties... etc. Why the luck stiff :-) How could beauty, happiness, motivation, pride, pleaseure, enthusiasm be involved in coding. Very funny counter examples for python and java go here... This all boils down to Misattribution. You have a physical reaction and you think that it has something to do with the mental plane but it doesn't. People on the right hand side of the movie have 10% greater ratings than the left.

Installing vim with +ruby or --enable-rubyinterp on OS X 10.5.2

2008-05-27T07:55:00Z

The problem: When installing vim from source with --enable-rubyinterp option you are receiving numerous unsolvable errors (such as ncurses cannot be found when it is installed and you are directing ./configure to it using --with-tlib ncurses). Or, you are trying to install vim +ruby using MacPorts and you are getting similar errors. The solution: install vim using MacPorts and install Ruby first.

sudo port install ruby
sudo port install vim +ruby +huge

Newly installed version is in /opt/local/var/macports/software/vim/7.1.270_0+huge+ruby/opt/local/bin After launching this version of vim it responds to:

:ruby puts "hello world!"

If you have the following in your .bash_profile or .profile file and you reopen your terminal, the newly installed MacPort will take precedence over the default OS X install of vim:

export PATH=/opt/local/bin:/opt/local/sbin:$PATH

Note that having your MacPort ruby install take precedence over your default ruby installation can modify your environment. Particularly, it may cause ruby to look for a different gem library. You can verify you are using the correct configuration by checking your gem path, version, and gems that you have installed:

which gem
gem -v
gem list --local

If your MacPorts gem version is taking precedence over other gem libraries that you would prefer to use, you can modify the file name or remove the version of the gem that you do not want to use. It is likely that MacPorts is referencing /opt/local/bin/gem and you would like it to reference /usr/bin/gem.

Fibonacci Sequence Recursion in Erlang

2008-02-16T20:00:00Z

Erlang uses recursion extensively - so lets do some classic "Fibonacci sequence":http://en.wikipedia.org/wiki/Fibonacci_number recursion code katas for Erlang. Note, that I define things a little differently than the Wikipedia example I just linked to. In the algorithms below, I have defined the first first two elements as = 1 and there is no N defined for 0. So: fib(1) = 1, fib(2)=1, and fib(N) where N > 2 = fib(N -1) + fib(N - 2). For fun, lets try a few different algorithmic approaches to this sequence. The classic example of Fibonacci sequence recursion is a literal representation of the algorithm, which ends up being rather computationally intensive. It looks rather elegant but starts getting massively slow around N = 35. This is because N is recomputed recursively every time it is used!

fib_slow(1) -> 1;
fib_slow(2) -> 1;
fib_slow(N) -> fib_slow(N-1) + fib_slow(N-2).

Here's an example of a different algorithm, where each value is accumulated in a list and looked up rather than recomputed. There also is a io:format print statement that returns the final sequence of N numbers. This method is a lot faster than the inefficient version above. You can set N = 1,000 without much fuss. If you comment out the print statement you can get to N = 20,000 speedily.

fib_list(1) -> io:format("~p~n",[[1]]);
fib_list(2) -> io:format("~p~n",[[1,1]]);
fib_list(N) when N > 2 -> 
	hd( fib_list1(N-1,[1,1]) ).

% delegate for numbers greater than 2
fib_list1(1,L) ->
	io:format("~p~n",[lists:reverse(L)]),
	L;
fib_list1(N,L) ->
	[H1,H2|_T] = L,
	fib_list1(N-1,[H1 + H2 | L]).

The final method computes the value but does not store it in a list. This one is the fastest (although I have not benchmarked them). This method also eliminates the risk of filling up memory with the list of previous numbers. N = 50,000 breezes by with this algorithm on my 2.2 Ghz Intel Dual Core MacBook Pro.

fib(1) -> 1;
fib(2) -> 1;
fib(N) when N > 2 -> fib1(N,1,1).

fib1(3,P1,P2) -> P1 + P2; 
fib1(N,P1,P2) ->
	fib1(N-1,P2, P1 + P2).

Here are all of the different approaches (in reverse orer) in a module.

-module(fib2).
-compile(export_all).
% get nth fibonacci no storage
% quite fast, get's nth number with no list storage

test() ->
	13 = fib(7),
	13 = fib_list(7),
	13 = fib_slow(7),
	horray.

fib(1) -> 1;
fib(2) -> 1;
fib(N) when N > 2 -> fib1(N,1,1).

fib1(3,P1,P2) -> P1 + P2; 
fib1(N,P1,P2) ->
	fib1(N-1,P2, P1 + P2).
	
% fib sequence made by accumulating values in a list
% pretty fast even at N = 20,000.
fib_list(1) -> io:format("~p~n",[[1]]);
fib_list(2) -> io:format("~p~n",[[1,1]]);
fib_list(N) when N > 2 -> 
	hd( fib_list1(N-1,[1,1]) ).

%delegate for numbers > 2
fib_list1(1,L) ->
	io:format("~p~n",[lists:reverse(L)]),
	L;
fib_list1(N,L) ->
	[H1,H2|_T] = L,
	fib_list1(N-1,[H1 + H2 | L]).

% ineficient recursive method for fibonacci computation
% very slow at N = 35
fib_slow(1) -> 1;
fib_slow(2) -> 1;
fib_slow(N) -> fib_slow(N-1) + fib_slow(N-2).

Erlang Pragmatic Studio - Day 3 Notes

2008-02-15T19:07:00Z

Erlang Pragmatic Studio with Joe Armstrong & Dave Thomas – Chicago, Feb 13-25, 2008.

Day 3 – Feb 14,2008.

Here are my notes from Day 3 of the Pragmatic Studio Erlang course. Note that these notes are only vaguely edited. Be aware that these are notes so there are certainly typos and errors.

Beginning of the Day

Always good to develop from the specific to the general. e.g. make a simple app, then parameterize it with functions. Universal server that evaluates F is a very powerful generalization of a specific application (e.g. messaging).

can send a message to your self:

self ! a. % Could tie into recursion

a lot of users Erlang guru’s (tailf, Kredita) don’t use OTP, as they participated in the OTP creation. But if you build up the Gen server from incremental steps, you may find it not so difficult. The question is, does OTP really fit the problem at hand.

If you do spawn_link from the shell, report errors back to the shell.

erl -bootstart sasle % spelling system application support library. Get error logger and a few other items.

sasle libraries were added into Erlang at a later date for historical reasons.

Bit Syntax

In ruby or other languages would be fairly tedious (ands, ors, bitshifts, etc). Erlang has a high level syntax that lets you specify the structure of binary data.

Red = 2, Green=61, Blue=20.
=> 20
Bin = <<red:5>>
=> <<23>>
io:format("~8.2.0B~8.2.0b~n", binary_to_list(Bin)).
=> 00010111 10110100

Bit Twiddling History:

Erlang had lots of protocols internally in Erickson. Decided to implement every RFC – but was too difficult. Lots of bit twiddling. Because of this they made a sub-language for doing this bit-twiddling so that this was implement any protocol. Optimal and efficient unpacking of datastructures.

1/3rd of details were in the Erlang book. The rest needs to be found in the man pages.

There are also bit comprehensions – generators of bit stream. Good for huffman encodings, MIPS processing, etc. Very efficient. This is in the experimental phase but will be integrated into the system. Undocumented items are not official and may be changed. Once the docs are written it’s official.

Erlang mailing list erlang.org > FAQs and mailing lists.

Book page 86 great shoutcast server for distributed audio. There is an excercise for bit processing, but we are skipping it in the interest of time.

file at a time I/O

Reading file at at time is about 10-20 times faster than reading line at a time. Erlang is not as efficient as some other languages for string matching. Use [H|T] to parse. Only multi-gig pattern are difficult for this. binary input and output is definitely fastest.

file:consult(File) -> {ok, Term} | {error, Why}

Term Access – read configuration file

-module(test4).
-
read_config() ->
    {ok,[{host,Host,port,Port}]} = file:consult(...)
    ...

Term IO

inefficient

io:read('enter a term').
enter a term > {hello, "joe"}
io:get_line(...) %read from a shell

erl_scan:string("abc,123,{hello,joe}"). % turn into erlang tokens - very efficient
% written with explicit recursion that is more efficient than regex

file:pread ...

map onto standard file manipulations:

file:list_file(Dir)
file:delete()
...

filename:split(FileName) => [Component]

DNS servers are actually fairly easy. In a real DNS server there is tree walking for the sub-domain, domain, and tld xxx.xxxx.xxx

lots of DNS servers have interesting traffic patterns… tie ins with advertising sites, etc.

Erlang doesn’t link code into it’s Kernal. Need to establish a socket for C libs. Send messages using binary.

Orientation:

nodes – when you say erl it starts a node. Doesn’t register this node by default.
distributed erlang consists of a number of nodes that know about eachother.
SMP takes advantage of multi-core – this is not distributed Erlang
Socket distribution – is also not distributed Erlang

same network: erl -sname %short name erl -name % different names

There is a magic cookie that they must share. Simple challenge request, traffic is not encrypted. Can communicate over SSH. DNS and name recognition systems are often poorly configured and this causes some difficulties for setting up distributed erlang apps. There are som FAQs on debugging distributed Erlang.

Three new primitives:

spawn(Node, Mod, Func, Args) – links work the same
alive(Node) – tell the system you are alive

Main libs:

rcp
global

Test on local nodes, then over a cluster. Dave: mnesias cool be cause you can query it using list comprehensions.

erl -sname dave

rpc:call('joe@Daves-Powerbook', erlang, node, []).
rpc:call('joe@Daves-Powerbook', erlang, exit, [kill]).
rpc:call('joe@Daves-Powerbook', erlang, halt, []).

Shell commands for interacting with erl nodes:

^g gets you into the shell
j – jobs
r ‘dave@Daves-Powerbook’ % open a remote shell
c 2 – change to the second process
now at (dave@Daves-Powerbook)>
toolbar:start(). % show current processes – queries OTP structure

Security:

Security on distributed Erlang is very course grain. It’s all access or nothing.
Perfect for a fire-walled corporate cluster.

Need to shore cookie to other machines. Check your home directory – .erlang.cookie

OTP – Open Telecomms Platform

This is the platform / framework for building Erlang applications.

Error logs
Hot swapping modules
Common tasks that everyone needs
etc.
Mnesia
SASL
SNMP Agents
Web Server

Lots of undocumented corners. Programming Erlang book is very helpful. It’s Open Source in the sense that you can see the source. But commit rights are controlled. It is product quality. There is no difference between Erickson’s products released and the Open Source distribution. There are 15 people working full time to maintain this – stable, battle tested.

OTP Principles

Joe Armstron PHD Thesis
erlang.org/doc/design_principles/part_frame.html
client-server gen_server
finite state machines gen_fsm
event handling gen_event
supervisor gen_sup
Applications – e.g. mnesia, standard libraries, collections of processes shipped in it’s own right. start and stop etc.
Releases – 4-5 applications combined form a product
Application upgrade – upgrade a running system

Finite state machines

State x Event -> State1 x Action
Way of writing pattern matching on state and event.

gen_event

no reply back (error log)
event to new state, no reply

Order of reading:

Joe’s thesis – best intro to OTP: sics.se/~joe/thesis
erlang.org/doc/design_principles/part_frame.html
forthcoming O’Reilly book

Behaviors:

OTP name for “design patterns”
Callback module

Case studies:

AXD301 – 1.6 million lines of Erlang, built using OTP.
Nortel Networks

Factoids:

gen_server is the most used behavior.
supervisor bridge – allows a C module to be fault tolerant in use with Erlang.
gen_server – A generic client-server model

Types of Supervision:

1:1 vs 1:N supervision
1:1 – if child dies only 1 is restarted
1:N – if challed dies all child sibling processes restarted

trapexit.org is a good site.

Distributed applications:

Must have the same beam code on all machines. There are checksums.
Must have the same version of Erlang.

One way of solving this is to have the share the same backend NFS system.

Explicit code distribution. Send the code in a message:

{ok, Bin} = file:read_file("xxx.beam"),
Term = {apply, erlang, load_module, [Mod, Bin]}),
gen_tcp:send(Socket, term_to_binary(Term))

run it:

{apply, xxx, start, [...]}

dynamic code upgrade:

Mod:Func(Args, ...) the latest version of Mod is called
If you reload Mod then you can run two versions of the code at the same time
You can only have two versions of the code running at the same time, an old and a new version
To load a third version you must call erlang:purge_module(Mod) before reloading the module (think of this as a two place shift register)
to load code call erlang:load_module(Mod, BeamCodeBin)

You can find all the number of processes running a particular version of a module. Some may never change, those you may have to kill.

When you load new code you get old and new code running at the same time. You must call purge_code before you add a third version of the code. This is going to effect 0.0x of people using erlang. After a crash you need to look at these issues.

This code swapping is not well documented. Mailing list may be helpful for this area.

Q & A With Joe Armstrong

What is the relationship between processes on the OS and Erlang? One process per whole Erlang instance on the OS.

When you spawn there is a run queue. The process that currently gets to the top of the queue. Every time you do a function call it’s called a reduction. If it’s waiting to receive a message, it’s taken out of the scheduling queue and is put into the waiting for message queue. If mail is delivered it’s put back in the run queue. There is no polling that is used. When you send a message to a processor and it hasn’t used all it’s reductions – may preempt …

Consider exposing schedule to Erlang? This would be very dangerous. But GC and as many things as possible should be written in Erlang.

What do you do in an app to seperate tests into a seperate file? Hierarchy of tests:

Unit test framework that comes from France (used by ejabber).

Erickson has a test server that is distributed with Erlang. Massive regression and unit tests.

Credita has modified to run test any time anything is checked into their repositories.

Quick check written by John Hues. Was written in Haskell has been ported to Erling. Generates random tests that satisfy certain properties. Does some automatic reduction of code execution to minimize error conditions. This is an expensive use.

E Unit – Dave found this somewhat cumbersom. Straight pattern matching was easier. Didn’t seem to have a huge value add. Would like to see how he can use differen’t naming conventions to automate and have some sort of runner. Find *.erl run anything named test and report if it doesn’t return a pass. Gen server seems easily testible by calling handler methods.

Diagraming conventions for Erlang? Not currently.

Petri Nets or activity diagrams?

Message sequence diagrams are used frequently. Joe is not a fan of drawing programs. Although, message sequence diagrams are extremely useful (Y = Time. X = Process). Joe is writing a program for animating Erlang.

erlang:trace(Pid, [Msgs, Call, list of things to trace]).

This process get’s lots of messages. Joe is writing a program that stores these in a file and then animates the activities of these processes over time. Working with a games developer to animate this.

Erlyweb?

Joe – hasn’t spent much time with this and YAWS (Yet Another Web Server).

Erlang graphic user interface?

Lots written. Tend to be hard to use – wswidgets being used for 3D process modeling. Interfaces to SDL, GTK, Cairo, etc. TCL library is the mainly supported one that works everywhere. Lots of people have been doing GUIs in web browsers – action script 3 and flex. Can also use flex2 as a device driver for Erlang – e.g. video stream. Makes it possible to make these applications more portable. Therefore can use Flash as a device driver. Drop dead beautiful – flash with AIR may be the way to go.

Redeployable packaging?

Erlang is packaged for win, linux, and mac. CEAN (Comprehensive Erlang Application …) ... Wings. Martin Logan, faxian. Martin Logan also has generator applications for Erlang (Erlware?).

Good frameworks for doing ontological modeling and reasoning?

Expert systems shell. University of Corona. Reported at the Erlang user conferences. Multi agent listener but not generalized to owl.

List of reference projects?

No canonical reference. But check trapexit.org.

Where is Erlang center of gravity?

Attracting lots of interest in Financials – especially in London. Finance industry targeted conference coming up in London. Much bank interest – extreme real time demands. Number of pure erlang based trading companies. Debt buying company – purchases debt and collect. Buys these debts a few moments after the transaction. Mnesia founder involved.

Future changes – where is it going?

Not many changes to the language. Millions of legacy lines of code needed for compatibility – syntax unlikely to change. But, implementation speed optimization and multi-processor support is likely to improve.

Embedded Erlang?

4MB probably the minimum size. Used to be 640k but may be a better domain for C.

String localization – UTF8?

In a sense it is solved. In another sense it is not solved. In Erlang there is no string type – it is a list of integers. UTF8 is an integer and can be stored. However, the interpreter from text to binary and back is not written. However, it is not difficult. It is in fact a library change.

What is a good candidate for Erlang?

Telecomm applications, lots of small processes. Individual processes have low ammount of short computations. Good at coordination, concurrency, switching. Not good at matrix multiplication, gif encoding etc. Financial applications fit the erlang profile.

Libraries for interfacing to SQL databases?

ODBC libraries. Could just open a socket to the DB. Java has a good arbitrary support for sockets. Check Yariv’s Erly web implementation.

Summary

Fault tolerance drives a lot of the design decisions behind Erlang. Simple functional language. Function selection is by pattern matching Variables are immutable

Concurrent Erlang

3 primitives: Spawn, Send, Receive – very simplified from Object Oriented model.
register / unregister can be used to associate a name with a process

Fault Tolerant

catch .. throw, try … catch … end
link, process_flag(trap_exit, true)

Distributed Erlang

spawn(Node, Mod, Func, Args)
or explicit term passing

Benefits of Erlang:

Technologies becoming more concurrent and people are looking for solutions to that probelm. Cooperative applications, etc. Erlang is a working solution. Not operating system dependent. 20 – 30 years of experience with fault tolerance measured in years.
Multi-Core ready
Processes in the language

Some projects to research:

AXD301 – biggest Erlang and Functional programming ever made 60 programmers x 3 years. Runs backbone of british telecom. Market leading in that sector. Peaked at 9×9s of reliability. This wasn’t a commercial project on 2-3% of the market.
Kreditor (Kreditor.se) – buys debt. #2 IP startup in sweden. $150 year. Self funded. Founded by alumni of Blue Tail.
SimpleDB (Amazon)
CouchDB (text db)
MociWeb (Mochimedia)
Ejabberd (jabber server – by extension Twitter?)
ErlyWeb (Erlang web framework inspired by Rails)