Monthly Archives: December 2015

Interactive Fiction

I don't know if this is the appropriate venue to do that, but I'd like to announce that I finished a stable implementation of a framework for developing Interactive Fiction in Pharo.

It is based on an old code written by Bob Jarvis for Dolphin, but it has
been changed so much that it is almost something new.

This is my first project and I'm still working around Metacello and
polishing stuff up. So I appreciate any code contributions, comments or
thoughts about it.

It is hosted on Smalltalkhub:
(http://smalltalkhub.com/#!/~ericvm/Smallworlds)

Gofer new
    package: ‘ConfigurationOfSmallworlds’;
    load.
((Smalltalk at: #ConfigurationOfSmallworlds) project version:’0.3′) load.
(AdventureShell world: (CDGameWorld new)) openWithSpec.

Eric Velten de Melo

Notes about Unicode effort

Today, we have a Unicode and CombinedCharacter class in Pharo, and there is different but similar Unicode code in Squeak. These are too simple (even though they might work, partially).
The scope of the original threads is way too wide: a new string type, normalisation, collation, being cross dialect, mixing all kinds of character and encoding definitions. All interesting, but not much will come out of it. But the point that we cannot leave proper text string handling to an outside library is indeed key.

That is why a couple of people in the Pharo community (myself included) started an experimental, proof of concept, prototype project, that aims to improve Unicode support. We will announce it to a wider public when we feel we have something to show for. The goal is in the first place to understand and implement the fundamental algorithms, starting with the 4 forms of Normalisation. But we’re working on collation/sorting too.
This work is of course being done for/in Pharo, using some of the facilities only available there. It probably won’t be difficult to port, but we can’t be bothered with probability right now.
What we started with is loading UCD data and making it available as a nice objects (30.000 of them).
So now you can do things like
$é unicodeCharacterData.
 => “U+00E9 LATIN SMALL LETTER E WITH ACUTE (LATIN SMALL LETTER E ACUTE)”
$é unicodeCharacterData uppercase asCharacter.
 => “$É”

$é unicodeCharacterData decompositionMapping.
 => “#(101 769)”
There is also a cool GT Inspector view:
Next we started implementing a normaliser. It was rather easy to get support for simpler languages going. The next code snippets use explicit code arrays, because copying decomposed diacritics to my mail client does not work (they get automatically composed), in a Pharo Workspace this does work nicely with plain strings. The higher numbers are the diacritics.
(normalizer decomposeString: ‘les élèves Français’) collect: #codePoint as: Array.

 => “#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 807 97 105 115)”

(normalizer decomposeString: ‘Düsseldorf Königsallee’) collect: #codePoint as: Array.

 => “#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 103 115 97 108 108 101 101)”

normalizer composeString: (#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 807 97 105 115) collect: #asCharacter as: String).

 => “‘les élèves Français'”

normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: String).
 => “‘Düsseldorf Königsallee'”

However, the real algorithm following the official specification (and other elements of Unicode that interact with it) is way more complicated (think about all those special languages/scripts out there). We’re focused on understanding/implementing that now.
Next, unit tests were added (of course). As well as a test that uses http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about 75.000 individual test cases to check conformance to the official Unicode Normalization specification.
Right now (with super cool hangul / jamo code by Henrik), we hit the following stats:
#testNFC 16998/18593 (91.42%)
#testNFD 16797/18593 (90.34%)
#testNFKC 13321/18593 (71.65%)
#testNFKD 16564/18593 (89.09%)
Way better than the naive implementations, but not yet there.
We are also experimenting and thinking a lot about how to best implement all this, trying out different models/ideas/apis/representations.
It will move slowly, but you will hear from us again in the coming weeks/months.
Sven
PS: Pharo developers with a good understanding of this subject area that want to help, let me know and we’ll put you in the loop. Hacking and specification reading are required 😉

Spur is coming

We will start migration to Spur today.
To complete it, we will require some time, specially to adapt the CI and check everything is ok. 

Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
- We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
- No more GC stops (noticeable when running large systems)
- We will be able to scale our systems up to 2G memory consumption without loosing performance.

But, this will have some drawbacks in the first times: 

1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
	- There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one. 

2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve: 
- there are some stuff not possible to compatibilise, notably: 
	- Structures now need to inherit from FFIExternalStructure
	- Arrays now are now shadowed
	- in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
	- THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases. 
2.1) ASMJIT will be removed from system and put in a separated packages. 
NOTE: There will be a blog post explaining FFI-NB architecture during the week.

3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)

4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.

5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).

Magritte Tutorial [Draft]

Hi guys

Here is the ci of the magritte tutorial that I’m revising and collecting material for

https://ci.inria.fr/pharo-contribution/view/Books/job/Magritte/lastSuccessfulBuild/artifact/book-result/BookletMagritte.pdf

If you want to add a chapter let me know.
I will add the ticketCorner and/or conference registration one in the future.

Stef

TinyBlog: a New Web Tutorial

Hi guys

with Olivier Auverlot we did several attempts (over the year) and we are happy to release a first draft of a web tutorial
around Pharo. We ask the user to build a simple blog engine.

It goes over
– Mongo
– Seaside
– Magritte

https://ci.inria.fr/pharo-contribution/job/TinyBlogTutorial/5/artifact/book-result/output.pdf

Soon Rest/Deployment/Garage?/NeoJSON/XML

If you want to participate (translate it to english) or improve it you are welcome.

We hope that it will serve as a basis to a lot of Seaside applications.

This tutorial will be used in the lecture I’m giving next weeks in Togo and will be edited as a lulu book.

Stef and Olivier.

Cog Rump Xen unikernel

 
I got curious about following up these two threads making Cog a unikernel...

Cog in the cloud
http://forum.world.st/Cog-in-the-cloud-td4796514.html

Hosting the Pharo VM on MirageOS
http://forum.world.st/Hosting-the-Pharo-VM-on-MirageOS-td4799424.html#none


One path forward is to use rumpkernel, which "provides a means to run
existing POSIX applications as unikernels on Xen"
https://blog.xenproject.org/2015/08/06/on-rump-kernels-and-the-rumprun-unikernel/

Deploying real-world software today as unikernels on Xen with Rumprun
http://events.linuxfoundation.org/sites/events/files/slides/xdps15-talk-final_0.pdf


So I thought I would try the "Building Rumprun Unikernels" tutorial here...
https://github.com/rumpkernel/wiki/wiki/Tutorial:-Building-Rumprun-Unikernels
which I report here in case it was interest.  The tutorial is based on
QEMU (and I think I saw a comment somewhere that Cog doesn't run so
well on QEMU and needs Bochs), so after that I adapted it from QEMU to
Xen.


64-BIT DEBIAN 8 JESSIE & XEN

So first I installed Debian 8 Jessie 64-bit on my new laptop
$ uname -a
...x86_64
Then installed Xen and after a reboot could choose it from the boot menu.
$ apt-get update
$ apt get install xen-system-amd64 libxen-dev

REBOOTED

Tested manual selection from menu to boot into Xen dom0.
Checked Xen was operating with...
$ xl info


32-BIT DEBIAN 8 JESSIE & XEN

However later I hit some complications determining whether the rump
unikernel was being compiled 32-bit or 64-bit in order to mix in with
32-bit Pharo, so to simplify things I did a complete reinstall of my
system as 32-bit.
$ uname -a
...686-pae
Then installed Xen again  Xen only has a 64-bit hypervisor, but this
was not immediately available under 32-bit OS.  I can't exactly
remember but I think I just needed to add multiarch before it would
find the Xen package.
$ apt-get update
$ apt-get install multiarch
$ apt get install xen-system-amd64 libxen-dev
Otherwise it may have needed something like "dpkg --add-architecture ..."

REBOOTED

First test, manually selected to boot into Xen dom0, after which as
root the following checked Xen was operating...
$ xl info

Then set it to automatically boot into Xen...
$ dpkg-divert --divert /etc/grub.d/08_linux_xen --rename
/etc/grub.d/20_linux_xen
$ update-grub
REBOOTED


====================================
DID THE QUICK START TUTORIAL
USING GENERIC HW ON QEMU

0. Installed support tools and libs

$ apt-get install build-essential git-core cmake

1. Built the rumprun platform

$ mkdir -p ~/Repos/rumprun-hw
$ cd ~/Repos/rumprun-hw
$ git clone http://repo.rumpkernel.org/rumprun
$ cd rumprun
$ git submodule update --init
$ CC=cc ./build-rr.sh hw
Took note of this output needed later...
toolchain tuple: i486-rumprun-netbsdelf
cc wrapper: i486-rumprun-netbsdelf-gcc
$ echo $(pwd)
    /Users/ben/Repos/rumpkernel-hw/rumprun
$ export PATH="${PATH}:$(pwd)/rumprun/bin"


2. Built a toy application

$ mkdir -p test
$ cd test
$ cat > helloer.c << EOF   ....
$ cc -o helloer helloer.c
$ i486-rumprun-netbsdelf-gcc -o helloer-rumprun helloer.c
$ rumprun-bake hw_generic helloer-rumprun.bin helloer-rumprun


3. Ran the toy application

$ su

$ rumprun qemu -i helloer-rumprun.bin

In another terminal, check its running
$ xl list
Name                                    ID  Mem  ...etc
Domain-0                              0    7583
rumprun-hellor-rumprun.bin  1        64
The app takes over the terminal its run from, so from another terminal...
$ xl destroy rumprun-hellor-rumprun.bin


====================================
ADAPTED THE QUICK START TUTORIAL
FOR XEN PARAVIRTUALIZATION

0. Started a new terminal to get an environment with a clean PATH

1. Built the rumprun platform

$ mkdir -p ~/Repos/rumprun-xen
$ cd ~/Repos/rumprun-xen
$ git clone http://repo.rumpkernel.org/rumprun
$ cd rumprun
$ git submodule update --init
$ CC=cc ./build-rr.sh xen   #<<<<
Took note of this output needed later...
toolchain tuple: i486-rumprun-netbsdelf
cc wrapper: i486-rumprun-netbsdelf-gcc
$ echo $(pwd)
    /Users/ben/Repos/rumpkernel-xen/rumprun
$ export PATH="${PATH}:$(pwd)/rumprun/bin"


2. Built a toy application

$ mkdir -p test
$ cd test
$ cat > helloer.c << EOF   ....
$ cc -o helloer helloer.c
$ i486-rumprun-netbsdelf-gcc -o helloer-rumprun helloer.c
$ rumprun-bake xen_pv helloer-rumprun.bin helloer-rumprun


3. Ran toy application

$ cd ..
$ su
$ export PATH="${PATH}:`pwd`/rumprun/bin
$ cd test
$ rumprun xen -i helloer-rumprun.bin
$ xl list
Name                                    ID  Mem  ...etc
Domain-0                              0    7583
rumprun-hellor-rumprun.bin  1        64
This took over that terminal, so from another terminal...
$ xl destroy rumprun-hellor-rumprun.bin


====================================
TRIED IT USING CMAKE

$ cat > CMakeLists.txt << EOF see this link for text...
https://github.com/rumpkernel/wiki/wiki/Tutorial:-Building-Rumprun-Unikernels

$ mkdir host_bld
$ cd host_bld
$ cmake -G "Unix Makefiles" ..
$ make
$ ./helloer

$ cd ..
$ mkdir rumprun_bld
$ cd rumprun_bld

Now with the given cmake line I got an error message:
   Could not find toolchain file:
../../rumprun/rumprun/share/x86_64-rumprun-netbsd-toolchain.cmake

$ find ../.. -name x86_64-rumprun-netbsd-toolchain.cmake
found it, so that the following worked...

$ cmake -G "Unix Makefiles"
-DCMAKE_TOOLCHAIN_FILE=../../rumprun/rumprun-x86_64/share/x86_64-rumprun-netbsd-toolchain.cmake
..
$ make
$ rumprun-bake xen_pv helloer.bin helloer
$ rumprun-bake hw_generic helloer.bin helloer
$ rumprun xen -i helloer.bin

Next I'll report on initial steps to get Cog working on Xen as a Rump
unikernel...

cheers -ben

Macro Recorder

Hello guys,

I would like to present you MacroRecorder, a tool to build custom source code transformations in Pharo.

The tool is available in the Catalog for some weeks now.
The current version runs in Pharo 5 and there is an old version running in Pharo 4.

To show how this tool works, I prepared a quick demo video.
Take a look: https://youtu.be/gW3l4O-WEGs

Right now, I’m interested in usage data.
That means, if you face a case of repetitive code transformation in practice, and I believe this happened to most of you, please:
(i) show me the transformations you did, in case they already happened in the past, or
(ii) consider using the tool to apply them automatically

And, of course, feel free to ask questions, provide feedback, etc.

Kind regards,


Gustavo Santos
http://gustavojss.github.io/

Material Design

Hi,

Kevin and me have worked this summer on a port of Material Design Lite for Seaside. It’s a work in progress but we think that someone could be interested to use and improve our work.

Best regards
Olivier