Interview for 3DVF

Founder and business developer Bernard Légaut talks to Benoît Rogez about Gamr7 and Ürban PAD for French 3D modeling website 3DVF.

Read the interview here (in French).

Add commentDecember 21st, 2009 claudia

Ürban PAD School Inaugurated

Gamr7 held its first workshop for aspiring and professional graphic and level designers on December 7 at the Gamagora headquarters in Lyon. The workshop, entitled Ürban PAD School, took place on the eve of this year’s Game Connection.

During the workshop, participants created procedural assets and edited a 3D city model using Ürban PAD. We explored the Maya – Ürban PAD pipeline by converting assets, got a feel for using the Editors, and used City Designer manual editing functions to make automatic changes to the city.

Thanks to Gamagora for hosting us, and to everyone who came out! We’ve included a few photos for your viewing pleasure.

Ürban PAD School

This Ürban PAD School was the first in a series designed to introduce the Ürban PAD concept and toolchain in academic and studio settings across the world. We’re busy putting together our travel schedule for 2010, with preliminary stops in Berlin and Montréal.

If you’d like more information on Ürban PAD School in your city, read the description below and take a minute to respond to the questions in the Comments section, or join the Facebook discussion.

What
Ürban PAD School is an intensive introduction to creating 3D city models with Ürban PAD, our procedural urban design toolset.

School participants work individually and in groups to create graphical urban content and city models. The format is laid-back, interactive, and collaborative, with emphasis on open Q & A sessions and guided tutorials.

Who

Ürban PAD School is for confirmed graphic and level designers (studio-affiliated and independent), as well as architects and anyone else who has design experience and is interested in learning a new toolset for creating detailed 3D city models.

Students are equally welcome. We specialize in creating workshops for students and instructors in specific focus areas (computer science, architecture, game design, et cetera).

How You Fit In
We’d be interested in knowing where YOU would like us to stop. Please take a few seconds to reply and let us know where you would be interested in attending or sponsoring an Ürban PAD School.

Bonus Question
Any pipelines (Maya to Ürban PAD and back again, or Ürban PAD to game engine) that you’d like to cover?

Add commentDecember 17th, 2009 claudia

Gamescom 09

Benjamin and Julien, Gamr7 developers and incognito City Punks, tell us about their week at Gamescom, procedural technology, and Cologne.

You arrive in Cologne at midnight after burning rubber through Luxembourg. In your right hand is a Gamescom pass. After a Kölsch or three and a good night’s sleep, what’s the first thing you do once you’re on the floor?

B: Have our pictures taken with the Raving Rabbids, of course.

Okay. Once you’ve escaped the clutches of the Rabbids, where do you go and what do you do?

J: We head out on the conference circuit.

What stood out?

J: There was a seminar about copy protection, which is interesting for us as we develop Ürban PAD 2.0. As the software evolves, it’s going to be more important to find solutions to protection issues.

B: We also had the chance to look around at some of the engines and think about what we could do, from a development standpoint, to make Ürban PAD integration easier for some of these engines. It’s useful to do benchmarking when you’re at this stage in development, because it helps you plan how to adjust your standards with respect to what’s on the market.

Thanks for bringing up Ürban PAD (you knew we would get around to it sooner or later). From a technical perspective, what kind of a feel did you get for the position of procedural technology in the current market?

J: The Crytek stand was impressive. They seem to be integrating procedural technology into their engine, and it’s interesting to see what’s coming out. They’re doing a significant amount of work with procedural textures and shaders. That’s the direction that Gamr7 is taking with larger-scale elements, like city design and procedural building construction.

B: We also got the chance to attend Dierk Ohlehrich’s presentation, Procedural Content Generation Taken to Extremes. He makes 4-5 kilobyte cities that are really fun and unique. We met a few other people who have made racing games and were impressed by we could do with Ürban PAD, given its small file sizes and rapid generation capacity.

So is Gamr7 reaching out to racers?

B: We’re reaching out to anyone who needs speed, flexibility, and artistic control – and appreciates the elegance of having them all in the same package. Everyone is welcome.

J: We were also encouraged by the direction that procedural technology seems to be taking. It’s still small, but it doesn’t appear to be evolving towards complete automation. People are starting to realize the time gains that can be made by going procedural. At the end of the day, however, it’s an aid and a timesaver that still has a human designer behind the controls.

Any predictions for how the domain will evolve over the next few years?

B: Graphics architecture itself is not going to evolve by leaps and bounds for another four years or so, until the next wave of consoles arrives.

J: That means that we’re in a period where individual artists and studios will be playing with procedural content generation as a way to enhance games in advance of the next big technological advance.

What would an industry Eureka moment look like?

B: Going back to what Julien was saying, I’m not sure that a Eureka moment is how the shift is going to take shape. What’s happening now is a gradual but steady realization that procedural technology is a viable alternative to manual processes. I think that it will really take off when enough people at all levels of production chains see that the results are graphically interesting and sophisticated, and that it speeds things up significantly. We saw a 45-minute generation demo on the Crytek engine, for example, that included some procedural operations.

J: Speed and good looks – those are the two key factors.

What else did you like about this year’s Gamescom?

J: Big conference rooms and a good vibe in Cologne.

B: It was interesting to see the advances that are happening in gameplay itself. Interfaces are really changing: you can play skater games while standing on a skateboard, or move your character onscreen by throwing and squeezing a ball. From a play perspective, games are getting a lot more interactive and even more exciting.

Any nightlife tips for developers who are down and out in Cologne?

J: Barbecues!

B: Kölsch is good, and cheap, as beers go.

I thought it was a traditional Cologner meal.

B: It can be…

See you in 6 months at the GDC in San Francisco!

Add commentAugust 31st, 2009 claudia

Declarative Cities: Creativity, Productivity, and Automation

What is declarative technology, and how does it provide solutions to production issues in current game pipelines? Lionel Barret, Gamr7 co-founder and CTO, talks about when and why a declarative approach can be useful.

The game industry is one example of a sector whose practices could benefit from using declarative technology, although the technology’s applications are not limited to this industry.

Lionel’s examples are based on many years of experience creating software for the game industry, most recently with Gamr7, a procedural software company based in Lyon, France. The company’s flagship product, Ürban PAD, is declarative software used for virtual city creation.

What is declarative technology?

Declarative technology is a way of getting a result from a program by telling it what you want rather than how you want it done. You specify what you want instead of how the process should take place in your computer. You’re giving an order and the computer does the work to supply it. You don’t care how the computer does the work.

A declarative approach presents several clear advantages.

As we’ve said, it describes the what instead of the how of a process. Because of that, the data is more flexible and faster to create. It is easier to change the request (what you want) and let the computer do the heavy lifting than to redo all the steps needed to create the data for a result that is sometimes only slightly different.

Another advantage is the small file size. A declarative file contains an order, a request and therefore is very small as it doesn’t contain the final data. For some applications (like MMOs), small file sizes are a huge asset as they are transmitted faster and therefore cost less. Note the declarative file needs to be processed on the client to get the final data, but this is often a very advantageous trade-off.

An additional advantage of declarative technology is the composability of its data. Data is composable. You can combine it without any effort. Composability of final data is often much harder.

The tipping point toward a declarative solution often occurs when the data set gets too big to manage or to create directly. You need some abstraction to manage/create it efficiently. Outside the game industry, SQL  is a good example of using a declarative solution to manage data.

Past this tipping point, the net loss of time and money outweighs the familiarity and stability of a conventional approach. Sifting through reams of data becomes an unjustifiable losing proposition in terms of resources. It’s easier to organize the data in small chunks that can be clustered in a descriptive fashion.

How does declarative technology apply to game production?

The game industry faces many challenges when it comes to producing virtual worlds. Short deadlines are among the most pressing, but there are others like achieving the right mix between new technology, fun and visuals.

In most games, a city has fifty thousand objects, minimum. During production, these fifty thousand objects will be usually thrown away three to five times. This happens for one of two reasons: bad feedback from the quality assurance team, or a publisher-imposed constraints that come near the end of production. (In huge game cities, like the one in Grand Theft Auto, only parts of the city are thrown away.)

Changing a city is very costly in terms of time and money, especially in the later stages of production. Even though this hurts quality, producers often choose not to do it: it is simply too late, according to the production schedule. This is an unfortunate but reasonable thing to do.

A declarative approach to city construction would alleviate or even make this rigidity disappear. Specifically, it can help on four fronts:

1. Prototyping : This is a step where the benefits of declarative technology, especially flexibility and small file size, are very much in evidence and a boon to exploring new gameplay/backgrounds. The flexibility of the technology means that if your first prototype isn’t what you wanted, you can simply refine your input and obtain a different result. If you decide that your existing road network isn’t good enough, you don’t have to create it all over again. Instead, you change the data parameters/input, which generates change in the appearance of the roads. The process will take several hours as opposed to several days or weeks.
2. Pitching (to a publisher): Using a declarative approach also allows you to give a demo with a bigger scope. Again, flexibility comes into play here: you profit from size independence. Using declarative technology, it’s as easy to make a city the size of Paris as it is to make one the size of New York by changing a single parameter and nothing else. You aren’t limited to just one prototype per data set – you can create multiple representations of what you’re trying to accomplish, and what your results could look like. This increases the possibilities and the attractiveness of your pitch.
3. Production: The advantages of a declarative approach for production are twofold. First, you are always able to re-use the data you’ve created for a prototype. This means that you can re-use part of a prototype, or the prototype as a whole. If you like the look of your urban content, but not the road network, you can re-use the data that created the urban content while changing the road data. Second, the flexibility of the data means that teams are able to produce bigger and better cities that can be reliably and massively changed up until the last minute. Because declarative technology allows you to take the prototype all the way to production, you’re never locked in.
4. Postrelease: Declarative technology allows easy re-use of the completed work. We’ve touched on this in the previous points: re-using data throughout the prototyping and production processes. However, the data can also be stocked and changed after production. This capability enriches the resulting database template library and means that data will be available for future use. This is yet another application of declarative technology’s flexibility.

Still, the obstacles to using declarative technology in content creation should not be overlooked. They’re twofold: pipeline stability and creative freedom.

The stability of the pipeline is important to the smooth progression of a project. Because the team is familiar with the tools and the processes, it can budget its production more efficiently. Even if pipeline function isn’t ideal, many producers prefer to lower the risks of the unexpected. This is a trade-off between efficiency and risk, and a quite subtle one to measure.

On the creative side, people are afraid of what we call the “sorcerer’s apprentice complex.” A fully automated declarative approach can get out of control or – more likely – go against the creative vision. The ability to fine tune anything in the game world is crucial to quality, and in easing artists’ fears that declarative automation will be an all-or-nothing proposition.

What advantages does Ürban PAD offer as a declarative solution to city construction?

To summarize: flexibility, speed, small file size, and composability.

Flexibility

Ürban PAD was designed to capitalize on the flexibility of declarative technology. This translates into data flexibility on both a local and global level.

On a local level, Ürban PAD’s activity parcel templates can be changed and diversified through using declarative data. Many different graphic objects can be created from one data specification:

On a global level, you can make micro- or macro-modifications to the declarative rules. For example, if you decide that you want more people in your city, you can change a single parameter to increase the number of people without wading through hours of manual placement. Artists are the gatekeepers here, dividing up the work and attacking it head-on. They can apply global rules to a zone, or use a template by itself. They can check templates against the automatically produced results. It’s also possible to apply global rules during automatic processing, and to a whole city:

Speed

Declarative technology allows Ürban PAD to carry out prototyping much faster than a human operator. As we saw in the example above, you can add thousands of people (or houses, or decorations) to a landscape with just a click of your mouse. Fifty thousand graphic objects can be created within ten minutes.

Small File Size

The small file size of declarative data enhances Ürban PAD’s flexibility and performance. The data is small, and easy to retrieve, use, and store for future projects. A city prototype (without the assets) represents only 100k worth of data.

Composability

Ürban PAD data is composable. The numerous graphic variations on one data set, as you see below, combine to produce cities of rich content and varied appearance. You can create a prototype for a house using a data set, give it four or five different appearances by programming change into the data set, and then combine the variations to create a city sector. Note that Ürban PAD can be used to create any object – triggers and sounds, for example – not only cities.

How does Ürban PAD address the challenges of pipeline security and the sorcerer’s apprentice problem?

Ürban PAD was created with both considerations in mind. Having worked in game development, we know that reliability is something that teams are not willing to compromise. At the same time, the progress that can be made with declarative technology has real benefits for budgets and production schedules. These benefits are going to be sought-after with the new industry developments, especially as gaming moves toward ever-richer content and online formats.

Pipelines and Ürban PAD

Ürban PAD works with any existing pipeline. You’ll simply add a new tool for city creation to your toolbox. Ürban PAD does not replace your pipeline, and is easily integrated without compromising your pipeline’s stability. The software allows any kind of workflow, and especially privileges parallel workflows. Graphic artists and level designers are able to work on different elements of the same project at the same time, so your team is never blocked by waiting for someone at another step in the production chain to finish a project.

Optional and Customizable Features

You can choose to integrate all the Ürban PAD elements into your pipeline, or simply select the ones you want. You never have to use anything you don’t want to use. Do you want to use Ürban PAD but limit automation for a game or pipeline-specific reason? You can.

Also, complete manual editing is a built-in response to the sorcerer’s apprentice problem. Full manual editing is available at any step in the process, so declarative automation never overtakes your vision for your project. You integrate the steps you do want and ignore those that you don’t.

Standards and Formats

Ürban PAD will never keep your data from you. Collada and XML import/export are both available, and there is no lock-in format. (We can also create custom import/export functionalities on request).

Describe Ürban PAD’s framework.

This diagram is a good summary:

How do you see the future of declarative technology?

Over the short term, it will certainly contribute to cost-cutting, which is currently its most tangible advantage.

Over the middle and long term, it will also push toward runtime content creation by creating parallel workflows within the coming next-wave architecture. In this way, it will address the problem of team sizes and amount of production work. Both increase exponentially past a certain point of gameplay length.

For example, team size increased fourfold with the arrival of the current generation. The next wave of development and innovation is underway, but the industry cannot physically support another personnel increase of this scale. Automation through declarative technology is a practical solution for controlling costs and team size.

(Illustration credit: http://lostgarden.com/2007_02_01_archive.html)

On the other hand, you have the arrival of next-wave architecture, which will be hugely parallel and could afford to process declarative data without impacting the other components of a game, such as display and sound. The convergence of these two phenomena – the need for solutions to the problem of development team size, and hardware that is built to accommodate and encourage parallelism – could be a “perfect storm” for the takeoff of procedural technology.

The real open-ended question is the coming of cloud gaming. Onlive is the first foray in this direction, with graphic computations moving to a remote data center. This will allow rich content to be delivered even faster, and if it’s successful, it’s something that could give procedural technology a push forward. We’re excited about how Ürban PAD could be adapted to a model like this. If graphics can be delivered this way, we can imagine that the technology will eventually evolve to handle triggers, sounds, and other vital components of gameplay. Ürban PAD can provide a declarative solution to the creation of these components, too – not only cities.

Using declarative technology to enhance the functioning and speed of existing pipelines is a relatively new concept that is making its way to the forefront of the industry. The technology is still evolving and adapting to industry needs, but we’re excited about the possibilities and some of the concrete applications that we’re starting to see.

Where can we get updates about Ürban PAD and Gamr7?

Just visit our website at http://www.gamr7.com. You can find our news feed, keep updated on the latest developments, or send us an e-mail at contact@gamr7.com.

Add commentJune 11th, 2009 claudia

Accessing Recursive Haskell Data Structures from C/Python

As promised we are going to explain how to access recursive Haskell data structures in C (and in combination with the previous post one should be able to also access these structures in Python). As a bonus we do it the other way around too.

However, before one can begin sending recursive data structures from Haskell to C and vice versa one first needs to be able to do it for non-recursive data structures as explained here. Go and read it!

In short, the Haskell FFI Addendum specifies an interface in the form of a type-class Storable. It states that when you call poke someptr haskell_value it will translate haskell_value to something the C Runtime System will understand. In the same manner, when you call peek someptr, it will construct a Haskell value by interpreting someptr in combination with the desired Haskell type and since this all is fairly dangerous, it happens in the IO monad.

There are some basic types for which the Storable instance in defined (e.g. Int), but for all user-defined types you need to do it yourself. There is currently no compiler to take a Haskell data type and turn it in readable C structs.

To make writing Storable instances somewhat easier there is a tool called hsc2hs which creates from a .hsc file containing macros a C source file, which when run creates a Haskell (.hs) source file. It is basically a program that takes into account the alignment of your data structures, so that you do not need to worry about it, since it uses offsetof (man 3 offsetof), which is a standard C macro defined in stddef.h.

The offsetof macro is also used in a macro to make the implementation of the alignment method a mechanical procedure:

#let alignment t = "%lu", (unsigned long)offsetof(struct {char x__; t (y__); }, y__)

To explain how it works we need a simple definition, quoting Wikipedia: “A memory address a, is said to be n-byte aligned when n is a power of two and a is a multiple of n bytes.”

Some basic machine types have certain alignment restrictions, or work faster within such restrictions. So, suppose a 32 bit int on a 32 bit architecture needs to be 4 byte aligned, then if we apply the alignment macro, the equation minimal is solved. This is a stronger property than what is required according to the GHC documentation for the alignment method.

There are other tools available, but all of them are written with the assumption that one has a C library that needs a Haskell binding. An ideal solution would be to have a compiler which takes a Haskell source file and generates for every data declaration a corresponding C struct definition (or whatever foreign language you are using) including Storable instances. All it needs is a naming convention and some time :)

Ok, so much for the background information. On to the code.

Meta-note: If you know a better WordPress plugin (this one (EasyLaTeX) does not interpret the EasyLaTeX version of the LaTeX $ x > 1 $ correctly), please comment.

First I will dump the C code clib.c:

#include <stdio.h>
#include <stdlib.h> /* to get malloc */
#include "clib.h"
 
/* O(Constant sweetness) space */
void print_list(TListElt * list)
{
  while (list->tag == LIST_CONS)  {
    printf("%s\n", __func__);
    printf("list->tag: %i\n", list->tag);
    printf("tag == LIST_CONS\n");
    printf("address of list: %p\n", list);
    struct Cons * f = list->elt.cons;
    printf("f: %p\n", f);
    printf("data: %d\n", f->data);
    list = f->next;
    printf("address of next list: %p\n", list);
    list = f->next;
  }
  if (list->tag == LIST_EMPTY) 
    {
      printf("list->tag: %i\n", list->tag);
    }
}
 
/* Generates a linear recursive process */
void recursive_print_list(TListElt * list)
{
 if (list != NULL)  {
  printf("%s\n", __func__);
  printf("list->tag: %i\n", list->tag);
  if (list->tag == LIST_CONS){
    printf("tag == LIST_CONS\n");
    printf("address of list: %p\n", list);
    struct Cons * f = list->elt.cons;
    printf("f: %p\n", f);
 
    TListElt * n = f->next;
    printf("data: %d\n", f->data);
 
    printf("address of n: %p\n", n);
    print_list(n);
    }
  }
}
 
void make_test_list()
{
  printf("Entered %s\n", __func__);
  TListElt second =
    {.tag = LIST_EMPTY
    };
 
  struct Cons thecons = {.data = 666, .next = &second};
 
  /* This syntax initalizes the first defined member of the union,
     if you want to initialize any other, you need to use a separate statement 
     (http://www.experts-exchange.com/Programming/Languages/CPP/Q_10120178.html)
  */
  TListElt first = {.tag = LIST_CONS, .elt = {&thecons}}; 
 
  print_list(&first);
  printf("Exiting %s\n", __func__);
}
 
TListElt * make_test_list_returning_list()
{
  printf("Entered %s\n", __func__);
  TListElt * second = malloc(sizeof(*second));
  TListElt * first = malloc (sizeof(*first));
  second->tag = LIST_EMPTY;
 
  struct Cons * thecons = malloc(sizeof(*thecons));
  thecons->data = 666;
  thecons->next = second;
 
  first->tag = LIST_CONS;
  first->elt.cons = thecons; 
 
  print_list(first);
  printf("Exiting %s\n", __func__);
  return first;
}

And the header:

/* Define a struct that is compatible with the Haskell representation  */
typedef struct {
    int a;
    int b;
} Foo;
 
void print_foo(Foo *);
 
void add_a(Foo *f);
 
enum ListTag {LIST_EMPTY, LIST_CONS};
 
struct Cons {
      int data;
      struct ListElt *next;
};
 
union UnionOfOneElement {
    // void empty; implicit
    struct Cons * cons;
  };
 
struct ListElt {
  enum ListTag tag;
  // use of union is redundant here, it's just to illustrate the general translation
  union UnionOfOneElement elt;
};
 
// just to illustrate we can also use TListElt instead of struct ListElt. 
typedef struct ListElt TListElt;

And the Haskell Main.hs (which is using all the stuff we define in the rest of this post):

module Main where
 
{- this is a Haskell program using C library functions
  and Haskell functions constructing values suitable for C. -}
import Data.List
import Foreign.Marshal.Utils
import Foreign.C
import Foreign.Ptr
import Foreign.Storable
import Foreign.Marshal
import Numeric
import ExportHaskellToCStruct
 
main = do
    ptr_to_list <- f_returning_list
    -- construct a Haskell list from a ptr to a C list
    haskell_list <- peek ptr_to_list 
    putStrLn (show haskell_list)
    -- construct a simple Haskell list
    let fairly_large_list = foldr MyCons MyNil [1..100]
    -- and let the C world print it
    printList fairly_large_list

From here this is a literate Haskell document (the hashes in the comments are being incorrectly processed by hsc2hs so that it will not work without some substitution) describing how to use Haskell values in C and vice versa. The former is not a popular thing to do, because it is a task for which many people expect shiny tools exist, but there are none and moreover bridging this gap requires knowledge of various languages; this is why we hope you will learn from this post. It is intended to be read by anyone who wants to be able to share data structures between Haskell and some foreign language. Also, the level of acquaintance with these topics from the reader is assumed to be low; I am trying to bring the use of the Foreign Function Interface (FFI) to a wider audience.

We will go through the code almost line by line. The very first line is to make the compiler accept foreign declarations. Then we define a Haskell module named ExportHaskellToCStruct and include “HsFFI.h”, because hsc2hs does not. HsFFI.h contains the interface needed to initialize the Run-Time System of The Glasgow Haskell Compiler, in short the GHC RTS. Furthermore, we include the user-defined clib.h header to get access to the various C symbols we use.

\begin{code}
{-# LANGUAGE ForeignFunctionInterface #-}
module ExportHaskellToCStruct where
import Data.List
import Foreign.Marshal.Utils
import Foreign.C
import Foreign.Ptr
import Foreign.Storable
import Foreign.Marshal
import Numeric
#include "HsFFI.h"
#include "clib.h"
\end{code}

Then we also include stddef.h, which is a standard C header containing the offsetof macro, and define alignment (as discussed before).

\begin{code}
#include <stddef.h>
#let alignment t = "%lu", (unsigned long)offsetof(struct {char x__; t (y__); }, y__)
\end{code}

Create corresponding Haskell values for the enumeration from C, first argument is Haskell type, second argument is possible wrapper constructor (e.g. newtype Wrapper = Wrapper Int), the rest of the arguments are the C constructors. In this case we have not defined the second argument (the ,, in the code). The exact symbols being generated are used later in the code and I will refer back to here. Every identifier preceded by a # in this source code, except for CPP directives, denotes something that will be processed by hsc2hs. hsc2hs either generates new identifers, like in the #enum case or it constructs an anonymous function that will for example access certain fields in a C struct (e.g. #peek).

\begin{code}
#enum Int,,LIST_EMPTY, LIST_CONS
\end{code}

This defines a simple recursive data structure, which happens to be a list. The type-synomym merely saves some typing.

\begin{code}
data MyList = MyNil | MyCons Int MyList deriving Show
type MyListPtr = Ptr MyList
\end{code}

Take the function print_list as defined in clib.h and make it available as f_print_list in Haskell.

\begin{code}
foreign import ccall "static clib.h print_list"
    f_print_list :: MyListPtr -> IO ()
\end{code}

Now we have landed at the start of the most important part of the code: the translation between Haskell and C objects. The pattern for sizeOf and alignment should be fairly boring. One thing that might be confusing is that these functions ignore their first argument. That is simply because the type signature of a class method should mention the type variable of the type class at least once. The system could have been designed in another way, but it is not. Sorry.

\begin{code}
instance Storable MyList where
 sizeOf _ = (#size struct ListElt)
 alignment _ = #{alignment TListElt}
\end{code}

The method peek takes as input a C value and constructs from it a Haskell value of the desired type in the IO monad. We need to define this mapping and in this particular peek implementation we are going to translate a pointer to a struct ListElt (== TListElt) containing CInts to a Haskell list containing Ints. We either return MyNil, in case the tag indicates LIST_EMPTY, which is written like listEmpty in Haskell terms, or a MyCons in case of a LIST_CONS (which is listCons in Haskell). Just stating the obvious, this is why we needed the #enum earlier. The code starts with some debugging information, which is nice so you can see what is happening.

\begin{code}
 peek ptr =
  do
    putStrLn "Trying to construct a Haskell value from the C one"
    putStrLn ("Before the peek at " ++ show ptr)
    tagInHaskell <- (#peek TListElt, tag) ptr
    putStrLn ("tagInHaskell: " ++ show tagInHaskell)
\end{code}

The first case checks whether the value we read in equals LIST_EMPTY in C, and if it is true we return the base case MyNil. The non-trivial case on which we explicitly check, is the listCons case.

\begin{code}
    if tagInHaskell == listEmpty
     then return MyNil
     else
      if tagInHaskell == listCons
      then do
       putStrLn ("We are going to cons...")
\end{code}

We find a pointer to the Cons structure and in this structure we inspect (peek) the data field and bind it to c_data. We also find where the next list is located and then call peek recursively so that we get the rest of the list. Then we just combine the two and we have a valid Haskell MyList value.

\begin{code}
       ptr_to_cons <- (#peek TListElt, elt) ptr
       putStrLn ("ptr_to_cons: " ++ show ptr_to_cons)
       c_data <- (#peek struct Cons, data) ptr_to_cons
       putStrLn $ "cdata: " ++ show c_data
       pointer_to_next_element <- (#peek struct Cons, next) ptr_to_cons
       putStrLn ("pointer_to_next_element: " ++ show pointer_to_next_element)
       haskell_list <- peek pointer_to_next_element
       return (MyCons c_data haskell_list)
       else
         -- this is being overly cautious  
           error "Impossible state"
\end{code}

Here we obtain an address (the ptr) and we need to somehow fill it with a C value; we are marshalling a Haskell value to C. The implementation is defined in poke_implementation_for_MyList.

\begin{code}
 poke ptr haskell_list = do
      poke_implementation_for_MyList ptr haskell_list
      haskell_value_that_should_be_equal_to_haskell_list <- peek ptr
      putStrLn (show haskell_value_that_should_be_equal_to_haskell_list)
      return ()
\end{code}

The poke implementation is going to be using O(1) stack space to run. The C value we require is a ptr to a TListElt. The first case is again MyNil.

\begin{code}
poke_implementation_for_MyList ptr haskell_list =
 case haskell_list of
  MyNil -> do
    putStrLn $ "Building a C value corresponding to MyNil at " ++ show ptr
\end{code}

We construct a poke function with #peek and set the tag field of the structure pointed to by ptr to listEmpty (which is LIST_EMPTY/0 in the C world).

\begin{code}
    (#poke TListElt, tag) ptr listEmpty
\end{code}

The other case is that we want to translate the MyCons case to C. We need to translate the MyCons constructor to a LIST_CONS with the haskell_data value and the rest of the list.
We set set the tag to be the LIST_CONS, verify it has been written correctly and then we somehow need to set the next field to some address. Since we do not have such an address, we need to obtain one with malloc. We will not free this memory in this code, but you can do it yourself if you want with free (either the C free or the Haskell Foreign.Marshal.Alloc.free). In C you need to specify how much memory you want, but the Haskell malloc does not take an argument. How does the RTS know what to do? In Haskell this is done by using the Storable type-class and a call to sizeOf in the implementation of malloc. In this case one could point (haha) out, that one is specifying the type, but look somewhat further and you see a call to malloc where the type is being deduced by the type-inferencer. We also set the data field to the haskell_data value.

\begin{code}
  MyCons haskell_data more_haskell_list -> do
    putStrLn $ "Building a C value corresponding to MyCons at " ++ show ptr
    (#poke TListElt, tag) ptr listCons
    tagJustWritten <- (#peek TListElt, tag) ptr
    putStrLn $ "Tag set: " ++ show (tagJustWritten `asTypeOf` listCons)

    -- first construct a struct Cons
    ptr_to_fcons_cell <- malloc::IO MyListPtr

    putStrLn $ "ptr_to_fcons_cell: " ++ show ptr_to_fcons_cell

    (#poke struct Cons, data) ptr_to_fcons_cell haskell_data
    should_contain_haskell_data <- (#peek struct Cons, data) ptr_to_fcons_cell
    putStrLn ("Value read: " ++ show (should_contain_haskell_data::CInt))
\end{code}

We let the elt union field point at the newly constructed Cons structure and we have the tail-recursive call to construct the rest of the list. A use of the #offset macro is also demonstrated, but is not used. (_offset_next_field is the offset of the next field in the Cons structure.) Finally we set the next field of Cons to the address of the allocated memory (this time the type is deduced by the compiler).

\begin{code}
    (#poke TListElt, elt) ptr ptr_to_fcons_cell
    let
        _offset_next_field = (#offset struct Cons, next)
    ptr_to_memory_that_will_contain_a_TListElt <- malloc
    (#poke struct Cons, next)
      ptr_to_fcons_cell
      ptr_to_memory_that_will_contain_a_TListElt
    poke_implementation_for_MyList
      ptr_to_memory_that_will_contain_a_TListElt more_haskell_list
\end{code}

These functions were used to see whether we could call C code, and indeed we can.

\begin{code}
foreign import ccall "static clib.h make_test_list"
        f_make_test_list :: IO()

foreign import ccall "static clib.h make_test_list_returning_list"
        f_returning_list :: IO MyListPtr
\end{code}

The function Foreign.Marshal.Utils.with takes a Haskell value of type T, and gives it to a Haskell function that expects a Ptr T (i.e., T * name in C notation) and then returns whatever value this Haskell function returned.

\begin{code}
printList my_list = with my_list f_print_list
\end{code}

This final piece of code just exports Haskell functionality to the rest of the world (e.g. C/Python) like we did before.

\begin{code}
foreign export ccall foo :: Int -> IO CInt

foo = return . genericLength .  f

f :: Int -> [Int]
f 0 = []
f n = n:(f (n-1))
\end{code}

Congratulations, you made it to the end. Your perseverance level has reached a new height. I hope you liked it.

Add commentDecember 22nd, 2008 ron

Python, Linkers and GHC (or: the evade-the-impossible game)

The plan as outlined in the previous post was fairly solid, but the road was more bumpy than expected.

It turns out that ctypes can only call libraries created by ld -shared and not ld -r (gcc can do both). The actual shared library for a given Haskell library is then obtained by collecting the linker command from ghc -v output while compiling in –make mode. This command will fail, since it cannot link to create a complete executable, because there is no main function. The part of the linker command upto and including -DDONT_WANT_WIN32_DLL_SUPPORT is removed, we keep the rest and add ld -r -o libhaskell.o before it, so we can get a partially (if anyone knows more about this than man ld states, please tell me. The ld source file containing the implementation is a 7KLOC monster. ) linked library. This way we obtain a shared library containing everything that the user-defined Haskell library requires, including a.o. the GHC RTS.

Note: using ghc --make with C source code files is likely to cause the infamous “the impossible happened”-error from GHC. I can recommend to use the following command at least for the Windows version of GHC 6.8.3: ghc -v -shared -o foo.dll ExportHaskellToC.o ExportHaskellToC_stub.o. Why 6.8.3 and not 6.10.1? The shared library support is broken in 6.10.1 on Windows.

Having obtained libhaskell.o in this way, which is a relocatable object, we can create a shared object from it by calling ld -o libWithHaskellAndCStuff.so -shared libhaskell.o OptionalCStuff.o. Now, we have something which ctypes can use.

Initially, the Python code would call the C code that initialized the GHC RTS, but we were also able to remove the entire C layer and thus doing the initialization in Python. The code below shows how it works:

from ctypes import *
import sys
import os
import platform
 
def main():
    # not tested on Windows 
    if platform.system() != 'Windows': 
        libhaskellandfoo = cdll.LoadLibrary(os.path.expanduser('~')+ 
           "/tryingtomakeasharedlibrary/libWithHaskell.so")
    else: 
        libhaskellandfoo = cdll.LoadLibrary("C:/tryingtomakeasharedlibrary/foo.dll")
    print("hi from Python")
    argc = byref(c_int(1))
 
    #expected this to work like in the C version (see below)
    #if you know why it doesn't work, please tell me. 
    #argvcontents = (c_char_p * 2)("POINTLESS")
    #argv = pointer(pointer(c_char()))
    #argv = argvcontents
 
#  char **argv;
#  char * foo[2] = {"Pointless"};
 
#  argv = foo;
 
#  // Initialize Haskell runtime
#  hs_init(&argc, &argv);
 
    # since the "expected" way didn't work, 
    # we create the data structure with primitives that do seem to work
    s = c_char_p("POINTLESS")
    arraycell = pointer(s)
    testobject = (POINTER(c_char_p) * 2) (arraycell)
 
    # only needed when using GHC compiler, might be different for other compilers
    libhaskellandfoo.hs_init(argc, byref(testobject))
    libhaskellandfoo.hs_add_root(libhaskellandfoo.__stginit_ExportHaskellToCStruct); 
 
    print "hello from Python with a Haskell value %d" % libhaskellandfoo.foo(5);
 
    libhaskellandfoo.hs_exit()
 
   # if you reinitialize the GHC RTS again (after it was shutdown) it _will_ crash, 
   # so don't do that.
   # documented here: http://hackage.haskell.org/trac/ghc/ticket/2863
   # uncomment the following line in case you just want to call a C function named foo_in_c 
   # libhaskellandfoo.foo_in_c() 
   # defined in your library
    print("hi from Python after RTS has been shutdown")
    sys.exit(0)
 
if __name__ == "__main__":
  main()

The Haskell FFI code is the easiest part of it all (adapted from GHC documentation):

\begin{code}
foreign export ccall foo :: Int -> IO CInt

foo = return . genericLength .  f

f :: Int -> [Int]
f 0 = []
f n = n:(f (n-1))
\end{code}

The only thing that might confuse is the ccall, which simply is the calling convention of the standard C compiler on the system (this works on both Linux and Windows). If you would be providing a callback for a procedure in the Win32 API taking a function pointer, then it would have been “stdcall” instead of “ccall”, but I have not actually used the Win32 API in this way, so I do not know whether it will work.

This concludes being able to call Haskell code from Python. Calling Python from Haskell with a suitable license should also be possible, by using the Python C API (but do not expect a post on this!). However, stay tuned for the next post, which will tell you all about sharing recursive data structures between Haskell and C/Python.

5 commentsDecember 22nd, 2008 ron

Bridging Python and Haskell

Let’s first start with a bit of background. I programmed a fair amount in the heavens of Haskell, and thus managed to stay away from the world of segmentation faults, all available via the Foreign Function Interface (FFI), quite successfully. That is, until my evil French boss destroyed my serene image of programming when he asked me to implement a proof-of-concept to send Haskell data structures to Python on both Windows and Linux; this with the goal of transitioning the Gamr7 codebase from Python to Haskell.

The plan was to first get the basics working, which is calling Haskell from C. Then to eliminate the C layer and use Python + ctypes to call a simple Haskell function. This all while switching between Linux (where the Haskell ecosystem actually looks like an ecosystem ;)) and Windows. Having achieved this, we could continue to move simple structs between Haskell and C and then finally recursive data structures.

Also note, that we are not going to be using GHC to compile the main binary, because there will not _be_ a main binary. There will just be the Python VM calling Haskell code. The common wisdom of using GHC to compile your Haskell + C project does not apply here. The next posts will explain how to do that.

4 commentsDecember 19th, 2008 ron

Procedural Content by Jason Booth

(archived here because his blog is closed and this blog post is damn good)

Danc recently blogged on Procedural Content, sparked by an article from Introversion on why ‘Content is Bad’. While I am a big fan of procedural techniques, I have to point out that I’ve seen them used incorrectly more times than not.

Procedural Techniques are hard to get correct – and the less human intervention required by the process, the harder they are to mold to a desired result. As such, I’ve come to look at them in a slightly different light than they are normally discussed:

Procedural technique is another name for tool

Photoshop provides us with a huge array of tools to work with. We have brushes, layers and selections. We don’t think of these as procedural content generation, but in fact, that’s exactly what they are. When we paint with a brush, we’re modifying many pixels at once using a procedure defined by the program code and a small piece of content; in this case, the data for the brush.

Through many operations, an artist can create a masterpiece that would be impossible to generate entirely using procedural techniques. This would be technically feasible to do one pixel at a time, but by providing a higher level of abstraction (brushes), we reduce the difficulty curve of reaching that level of mastery immensely.
As such, it’s important to think of your procedural technique as a way to augment your creative process, either by creating a useful and understandable abstraction in the way a brush does for an artist – or by replacing large sections of ‘busy work’ with a procedure that fills out that data.

If your artist is afraid, so should you be…

Programmers love procedural techniques – artists (or designers) generally fear them. If a procedural technique is integrated correctly into your design, an artist will not see it as a thing that replaces their work, but rather helps them achieve their goals faster. In essence, a good procedural system is viewed as a tool instead of a procedural system.

The reason for this is quite simple; a tool has an understandable need, use, result, and a process which can be directed to solve a problem. If you can’t identify all of these for your procedural system, should you be building it?

Don’t over-solve the problem

Often when I hear people talk about procedural techniques, it’s as a way to ‘generate infinite content’; as if that is going to make your game last forever. But people are incredibly adept at seeing past the algorithm and sensing the underlying possibility space, and tiring of its parameters. As such, there is no such thing as infinite content.

Procedural techniques, and tools in general, should be designed to allow their users to create the right amount of content. If a tool is hard to use, people will avoid it and you won’t have enough of that type of content. However, the converse can be just as bad. If a tool is too easy to use, you will see too much of the type of content it creates, and users will tire of its output quickly.

For instance, when we designed the landscape system for Asheron’s Call 1, we designed it to allow an artist to quickly create large areas of terrain, and populate them with monsters, scenery, and ambient sounds. As such, we ended up with a huge world filled with this data. However, our process for placing hand crafted content was not nearly as refined. So while there were huge areas of procedural content for users to explore, there was a lack of the artists touch in many areas.

Your content is usually not, actually, procedurally generated

Often people talk about the procedurally generated dungeons of Diablo as content; but in reality, they aren’t actually content from the perspective of the design. You don’t actually explore dungeons in Diablo – what you’re really exploring is the treasure system. And while the treasure system does employ some procedural techniques, the real rewards of that system were entirely hand crafted.

In fact, the true success of Diablo’s procedural systems is that they managed to acceptably scale out a small amount of custom created content over a large amount of procedural noise. Thus, the ratio of hours played to custom content created is very high.

Use procedures to augment artist input, and stretch your content further

Guitar Hero uses a procedural animation system to animate the characters playing their instruments. It was written in about 2 weeks, and requires a very small amount of unique authoring input to work. It pulls most of its animation from data that’s already there for game play reasons, mainly the note gems. A small amount of data is required by the person authoring the song, and they have various hints they can give to the system to adjust the results.

We’ve improved that system over the last few years, and the data mining is becoming quite complex, as are the modifications to the animations and other data it’s outputting. But for the most part, we’ve avoided authoring extra data, because the carefully crafted note charts and meta-data are a goldmine for information about what the music is doing.

Reach your quality bar first

Before building a procedural technique, it’s important to understand all the nuances of what you’re trying to do. As such, it’s best to create a quality result before trying to break that down into some type of procedural system. Looking at the data, be it animation or content, will help you spot the patterns which can be replicated and varied, or the busy work involved which can be automated or abstracted. Attempt to solve the problem as a tools issue until it has been proven to be one that has to be solved in a generative way.

posted by Jason Booth

Add commentJuly 1st, 2008 lionel barret

Welcome

A blog is a strange thing the first time you approach it.
It is a the same time a tool to focus your thought and a discussion with the (internet) world.
So be it ! let pull down the lever, let the electricy run down from the sky to our strange creature and give it life !

The topic of the is blog are game production and game development. In our subterranean high-tech complex, we are building a software for the game industry. For obvious reasons, we won’t talk much about it specifically, but the blog will be our repository for thought around this ultra-secret world-domination device (including programming, marketing and strategy).

Add commentJuly 1st, 2008 lionel barret