Monthly Archives: April 2016

Web Browser Spring cleaning

Hi,

Some of you may know or use "ExternalWebbrowser" (maintained in the past by me, started back in the times right after Squeak and 
based on FFI), also there was an "ExternalWebbrowser2" fork from Sean later. First project got outdated with transition to NB, 
the second one stopped working for Pharo 5 due to the transition from NativeBoost (NB) to Universal Foreign Function Interface (UFFI).

Two projects and now the broken code created a little bit a mess again in finding an easy way in opening an external webbrowser
from Pharo. Updating my "ExternalWebbrowser" would easily bring confusion to newbies who thing "ExternalWebbrowser2" 
is newer. Having a an "ExternalWebbrowser3" would be silly too. 

So I decided to fix all this by providing a general "WebBrowser" package which now should solve the issue of easily opening an 
external webbrowser once and for all for primary platforms (Unix/Mac/Windows). The simple design allows to add also other 
platforms later if necessary. 

I based it for Pharo 4 on NB and Pharo 5 onwards on the new UFFI - so YES you can use it in both Pharo versions uniformely and
the same way - which should simplify things now again. 


# Project page/Repository

http://www.smalltalkhub.com/#!/~TorstenBergmann/WebBrowser

# Description

General and platform independent WebBrowser access for Pharo 4.0 onwards based on NativeBoost, 
for Pharo 5.0 onwards based on UFFI

- Mac
- Unix (Linux flavours)
- Windows

# Usage

	WebBrowser openOn: 'http://www.pharo.org'

# Installation

You can load it either from Configuration Browser (Pharo 4),  Catalog/directly by typing "WebBrowser" into Spotter (Pharo 5 onwards)
or with the following load expression:

	Metacello new
		smalltalkhubUser: 'TorstenBergmann' project: 'WebBrowser';
		configuration: 'WebBrowser';
		load

# Other

If you want to reference it in own configs note that version 0.1.0 is for Pharo 4 while 0.2.0 and following is for Pharo 5 onwards.

Bye
T.

Advertisements

A weak/leak story

Hi list,

With Pavel and Christophe we spent some time digging these last weeks chasing the memory leaks we were seeing lately. It is a long story to tell, so this mail is divided in three:

1) A brief intro to weak structures and finalization in Pharo, for those that do not know,
2) A bit of history to explain what happened in pre-spur and post-spur,
3) The actual cause of the memory leak today,
4) How to avoid them in your application, and what are we going to do to prevent this in the future.

For those that need/want/prefer just the practical explanation, you can jump over 2) and just read 1) and 3).

1. A weak explanation

To cleanup objects upon garbage collection, Pharo and Squeak use a finalization mechanism based on a Weak Registry. That is, if you want to execute some cleanup (like closing a file) when an object is about to be collected, you have to put your object inside the weak registry with the corresponding executor/finalizer object. The object you want to ‘track’ is hold weakly by this weak registry i.e., if the only reference to the object is from the weak registry, it will be chosen for garbage collection. When this object is collected, a special process in the Pharo image will send #finalize to your executor object where you implement your cleanup.

To interact with the weak registry, there are two main subscription messages:
– #add:executor: Will add an object to the registry with the executor that is send as argument.

– #add: Will add an object to the registry, and use as executor a ‘shallow copy’ of the object.

Some conclusions to be made from this:
1) If the executor points strongly to the object that we want to collect, it will never be collected. That is why the #add: message creates a copy of the object.
2) If we do not provide an explicit executor, the registered object should already contain all information required for the finalization (like file handlers or external pointers). If not, the shallow copy will not be able to finalize correctly.

Also:
– Using weak objects/references do not guarantee that #finalize will be called, you need to put your object inside the registry!
– Using weak objects/references do not guarantee that your object will be magically collected. You can still cause memory leaks!

2. A weak story

Pharo and Squeak use historically the weak registry mentioned above. Because of the limitations that we mentioned, a different kind of weak structure called Ephemerons is required/more useful. To overcome some of these limitations, Igor (Hi Igor! maybe you’re reading :)) implemented a couple of years ago a new finalization mechanism that, IIANM, worked as follows:
– Some weak objects could have a first instance variable with a special linked list
– When the object was about to be collected, instead it was removed from the weak structure and put into its container’s linked list
– On the image side, a special process iterated all special linked lists and executed #finalize on the weak objects.

This mechanism was called NewFinalization, in contrast with what was called LegacyFinalization. Of course these names are context dependent, since today’s Pharo is back to the so called legacy one ;). NewFinalization was implemented as the default finalization mechanim in Pharo, both in VM and image side. But the VM changes remained in the Pharo branch of development. After some discussions, I remember Igor and Eliot agreed that what they actually needed were Ephemerons, and since Eliot had started working on Spur at that time, he said he would provide Ephemeric classes with the new object format.

Basically, for those interested, an ephemeron is an association

weak key -> strong value

with the special quality that upon garbage collection all references to the weak key that are computed from the strong value (directly or indirectly) are taken as weak. This allows the collection of the weak key even if the strong value points to it, but requires some more machinery in the GC/VM. You can read more in here [1].

Until a couple of months/weeks ago, Pharo was using the NewFinalization mechanism with its special image and VM support. And Squeak was using the ‘Legacy’ one. And then Spur arrived.

So Spur arrived, and Eliot and Esteban made a lot of effort to simplify the VM’s maintenance, and they merged both branches. As a conclusion, Pharo Spur VM did not support any more NewFinalization. This provoked at first some leaks because objects were not being finalized. A couple of weeks ago, we migrated back the image code to use the ‘Legacy’ mechanism, see issue 17537 [2].

And then finalization was not working either. Nor #finalize was being called on executors, nor objects in the weak registry were collected. As a symptom, opening any tool will cause 30 new everlasting registrations into the weakregistry, and no tools were collected.

3. The cause
After lots of digging, we finally found what was the particular issue causing objects in the weak registry to not be collected. In some words, it is caused by the normal belief that “weak objects are magical”, which caused that weak references and finalizers are really spread over the system with no proper care. And particularly related to the usage of announcements.

To explain better, I made some pictures for you 🙂

***First, imagine you have a morph with its own local announcer. You subscribe to two events, and the graph will look like this.

 

strong.png

 

– the announcer knows two strong subscriptions
– the subscriptions know the announcer to be able to unregister
– the subscriptions know the registered object to send the message in case the event happens

This forms a closed graph that will be collected. No problem so far.

***Second, let’s see what happens if we use weak subsriptions:

weak.png

– the announcer knows two weak subscriptions
– these weak subscriptions know the announcer strongly to be able to unregister
– they also know the subscriber object but weakly
– THE difference is made by the weak registry: a global object that manages when and how objects are finalized. In the case of announcers, the weak registry will store weakly the subscriber morph, and strongly the weak announcer subscription.

So far so good also: the references to the morph are weak. When the morph is collected, the weak registry will execute finalize on the announcement subscriptions. The subscriptions will unregister from the morph.

***The really problematic case is the third one: mixing weak and strong subscriptions in the same announcer.

both.png

 

The object graph is just a mixture of the two other ones. One weak subscription and one strong subscription. BUT:

– there is a strong path from a global object (the weak registry) to the subscriber (the morph)
– then the morph is never collected
– the weak registry never finalizes the weak announcement subscription
– the graph remains there forever.

And these are the simple cases that show the problem. Imagine that you can have this same configuration but in cycles/chains among different morphs/announcements. Plus this is aggravated by evil globals (e.g., the theme and the HandMorph remembers the last focused morph, the system window class remembers the last top window even if it was closed…).

4. The solution?

Our solution for the moment is simple. We would like to enforce the following two rules for announcements:

– announcers local to a morph should only be used strongly. YES, this may cause small hiccups and leaks, for example if you register a morph A to the announcer to another morph B. But in the long term, these two will form a closed graph and will be collected.

– announcers used globally, such as the System announcer, should be used only and uniquely in a weak manner. Like that we ensure that they are loosely coupled for real.

So, please, please, do not use weak announcements unless you’re really sure of what you’re doing. At least, until we have ephemerons and we are sure everything works as expected. Ephemerons would solve this in a more natural way: if we model the weak registry subscription as an ephemeron, any reference to the weak #key that arrives from the #value will be treated as weak also.

Other action points we are working on:
– fixing tools to follow the rules above
– We are also writing tests to check that tools (gt*, Nautilus, Rubric, FT) do not leak.
– chasing other small memory leaks created by stepping, focus global variables…

((fogbugz allIssues select: [ :each | each relatedToLeak ])
flatCollect: [ :each | each participants ])
do: #thanks

[1] https://en.wikipedia.org/wiki/Ephemeron
[2] https://pharo.fogbugz.com/f/cases/17537/SystemAnnouncer-has-far-too-many-subscriptions

 

Parrot AR Drone 2.0 with Pharo

Hi, my name is Carolina Hernandez and I’m a Computer Engineering student at the University of Chile.
As part of my final degree project, I’m creating a communication API for controlling the Parrot AR.Drone 2.0 from Pharo. I’ve just released an early version. The code is hosted in Smalltalkhub:
I also made a video of a test flight, where the AR.Drone is controlled using the API and keyboard:
I couldn’t find any direct way to use a joystick in Pharo, if someone knows about one, I’d like to integrate it to my code.
Development and tests have been done in Pharo 4.0 and ubuntu 12.04
Comments are welcome!
Caro

Mocketry 30 for Pharo

I am glad to present simplest mocks for Pharo.
In this version Mocketry is dramatically improved and changed.  I try to provide most simple way to stub any message to any object and to verify any occurred behaviour.
Here is short example:
mock := Mock new.
mock stub someMessage willReturn: 1000.
mock someMessage should be: 1000.
mock should got someMessage.
mock should not got anotherMessage.
rect := 0@0 corner: 2@3.
rect stub width willReturn: 1000.
rect area should be: 3000 “area = width * height”.
rect should got width.

Pomodoro timer enhanced

I did a new pass on the Pomodoro timer for Pharo 5:

- settings with custom start value and custom colors
- better resizing behavior of the morph
- pause/continue - start/restart possibility
- default colors aligned with Pharo scheme
- pause state in Pomodoro instead of Morph
- separate core and tests in own packages

all included in a new release 1.6.

Just use Spotter/Catalog to load.

Enjoy the new video:  http://youtu.be/voSUzOu5feU

Bye
T.

Easy deploy of pharo applications on the web (linux)

From http://www.smallworks.eu/web/blog

(For this instructions, I assume you are using Pharo 4.0)

You need to put you application somewere, isn’t? So, easiest way is to install your application along your image/changes files:

1. copy you application .image and .changes file into the place you are going to run your web application (usually /var/www/YourApplication)

2. in the same place, do:

 $ wget -O- get.pharo.org/vm40 | bash

3. create a startup script, something like this:

File name: run.sh

sudo -u pharo nohup ./pharo /var/www/YourApplication/YourApplication.image --no-quit &

(Notice that we use an alternative user to run app. Is never recommendable to use root permissions. Remember you need also to change permission of all “YouApplication” subdirectory)

… and thats all :). This is enough to have your application running.

4. [OPTIONAL] Startup script with initialisation

Sometimes you need to prepare some things that might change (for example, running ports, database hosts, etc.).

For this the easiest way is to modify your run.sh script:

sudo -u pharo nohup ./pharo /var/www/YourApplication/YourApplication.image eval --no-quit "
	"Start web server"
	ZnZincServerAdaptor startOn: 8080.
	"Start Voyage-MongoDB"
	(VOMongoRepository host: 'localhost' database: 'myApplication') enableSingleton." &	

Notice that is very easy to adapt this to receive ports, etc. as an script parameter (in case you need to scale your application and use several ports, for example).

5. [OPTIONAL] NGINX configuration

I do not use Apache since some years, so I do not remember how this is done (but is doable, of course). This is a basic config that works:

 server {
	listen       80;
	server_name  myApplication.com;

	location  / {
		proxy_pass http://127.0.0.1:8080/;
		proxy_redirect default;
		proxy_set_header  X-Real-IP  $remote_addr;
        	proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;

		proxy_set_header Host $host;
		proxy_pass_request_headers on;
	}
}

And that’s is 🙂

OS-Linux-CentOS

Hi,

the OS-xxx series with (so far)

 - OS-Windows
 - OS-Linux-Ubuntu
 - OS-Unix
 - OS-Raspbian

now has a new member called "OS-Linux-CentOS" which is more or less an adopted/customized
pendant to "OS-Linux-Ubuntu".

Project page is at  http://www.smalltalkhub.com/#!/~OS/OS-Linux-CentOS

Ready for Pharo 5. As ever you can load from Catalog tool or directly by entering "CentOS"
into the Spotter search tool. 

Enjoy the video:

   https://www.youtube.com/watch?v=nAwtVeoll_k

Bye
T.

OS-Windows 2.1 project updated for Pharo 5

Hi,

the "OS-Windows" package [1] from my "OS-xxx" Pharo goodies series [2] is providing better access 
to native Windows APIs and wraps Windows functionality for easier access/coding in Pharo (browser, console, 
processes, shell, window, GDI, ...). Some of you may know or use this package already. 

Up to the last provided stable version for Pharo 4 all my code and bindings were based on "old" 
NativeBoost (NB). With the transition away from NB over to new UniversalFFI (UFFI) within Pharo 5 
it was necessary to port/migrate the code over to UFFI for all the OS-xxx projects.  

Work of the Windows port started when UFFI was still shaky (December) and in a separate repo [3]. 
It was not easy in the beginning, some headbanging against the wall and feeling like a crash test 
dummy was involved - basing the port on early UFFI code and without much documentation. 
Several UFFI issues had to be discussed with Esteban Lorenzano in private conversations on #slack: often 
leading to some updated UFFI versions afterwards and an updated worklog of Esteban [4]. 

In the Pharo spirit "step by step" all of this got analyzed, fixed, renamed, improved and integrated into 
latest Pharo 5 in the last months. Esteban was VERY VERY supportive and adopted UFFI for all the needs!  

    ONCE AGAIN: A BIG THANK YOU, ESTEBAN!!!

Long story short: I can now release "OS-Windows" again into the wild for Pharo 5: fully compatible, 
                  again with cleaner code, some more functionality and even more tests (from 77 to now 81)

As ever it is easy to access: open spotter in Pharo 5 and load "OSWindows". Full docu is on the project page [1],
the new version is fully compatible to the Pharo 4 version so there was no need to change a single 
line in the tutorial. ;)

Have fun
T.

[1] http://www.smalltalkhub.com/#!/~OS/OS-Windows
[2] http://www.smalltalkhub.com/#!/~OS
[3] http://www.smalltalkhub.com/#!/~TorstenBergmann/NOS
[4] http://log.smallworks.eu/


Deprecation Assistant

Hi everyone,

last week I’ve created a little tool that automates deprecation resolution. The goal is to provide a help based on the existing code (no need for extra actions). You can read more here: http://blog.yuriy.tymch.uk/2016/03/deprecationassistant-aiding-developers.html
You can install and try it out with:
Metacello new
   repository: 'github://Uko/DeprecationAssistant';
   baseline: 'DeprecationAssistant';
   load
This will add additional actions to the debugger once you will get a deprecation warning.
Have fun!
Cheers.
Uko