Dynamically composing languages in a modular way: supporting C extensions for dynamic languages

Many dynamic languages such as Ruby, Python and Perl offer some kind of functionality for writing parts of applications in a lower-level language such as C. These C extension modules are usually written against the API of an interpreter, which provides access to the higher-level language's internal data structures. Alternative implementations of the high-level languages often do not support such C extensions because implementing the same API as in the original implementations is complicated and limits performance. In this paper we describe a novel approach for modular composition of languages that allows dynamic languages to support C extensions through interpretation. We propose a flexible and reusable cross-language mechanism that allows composing multiple language interpreters, which run on the same VM and share the same form of intermediate representation - in this case abstract syntax trees. This mechanism allows us to efficiently exchange runtime data across different interpreters and also enables the dynamic compiler of the host VM to inline and optimize programs across multiple language boundaries. We evaluate our approach by composing a Ruby interpreter with a C interpreter. We run existing Ruby C extensions and show how our system executes combined Ruby and C modules on average over 3x faster than the conventional implementation of Ruby with native C extensions, and on average over 20x faster than an existing alternate Ruby implementation on the JVM (JRuby) calling compiled C extensions through a bridge interface. We demonstrate that cross-language inlining, which is not possible with native code, is performance-critical by showing how speedup is reduced by around 50% when it is disabled.